acceptodds
Under review as a conference paper at ICLR 2027

Benchmarking Fairness in Spiking Neural Networks: Data Bias and Spurious Features

Abstract

Evaluating fairness in Spiking Neural Networks (SNNs) demands rigorous benchmarks that reflect real-world complexities, yet existing assessments remain limited by superficial dataset diversity. This work introduces the first systematic fairness benchmark for SNNs, addressing two critical dimensions of realism: () demographic coverage gaps in training data and () spurious feature leakage (e.g., skin tone as a proxy for class labels). Our framework integrates four cross-demographic datasets together with a saturation-based cleaning protocol that yields paired original and cleaned variants, enabling controlled comparison across data conditions. Standardized evaluations of state-of-the-art SNNs reveal pervasive fairness deficiencies, including stark disparities between demographic groups and systematic shortcut learning driven by spurious low-level cues. Our code is available at: https://anonymous.4open.science/r/SNN-Benchmarks-8017.

Primary area:
datasets and benchmarks
Keywords:
fairness, spiking neural networks, benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_CCaaOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper presents a benchmark for evaluating fairness in SNNs on facial image datasets. It evaluates gender classification with race as the sensitive attribute on three datasets and examines group disparities in race-classification accuracy on RFW. The authors also construct RFW-Cleaned to compare results before and after color-based filtering and analyze group-wise learning trajectories. The results show that overall accuracy does not adequately capture group disparities and that the effects of filtering depend on the model configuration.

Critical strengths
  1. The paper addresses the relatively underexplored topic of SNN fairness and covers multiple model families and datasets, providing a useful benchmarking reference.
  2. Reporting accuracy, SP, and EO together demonstrates that different metrics can produce different model rankings.
  3. The analysis extends beyond final performance to group-wise learning trajectories and acknowledges that filtering does not benefit every configuration.
Critical weaknesses
  1. The citation supporting the research-gap claim appears mismatched. Altamirano et al. (2025) reviews algorithmic bias in public-health machine learning and does not appear to substantiate the claim that fairness is largely overlooked in SNNs. Moreover, recent work has already examined fairness in SNNs, including Perdigão et al. (2025), CalibraSNN: Fair and Calibrated Convolutional Spiking Neural Network for High-Stakes Industry Applications; Nasif et al. (2026), Reinforcement-guided hyper-heuristic hyperparameter optimization for fair and explainable spiking neural network-based financial fraud detection; and Yang et al. (2026), Towards Fair Spiking Graph Neural Networks via Debiased Representation Learning and Fairness-Aware Bernoulli Encoding. The authors should provide an SNN-focused literature review, qualify the “largely overlooked” claim, and clarify how their benchmark differs from existing work, particularly in relation to the claimed novelty as the “first systematic fairness benchmark.”

  2. The filtering experiment does not isolate the effects of color. Comparing the original and cleaned datasets changes both the training and evaluation samples. Performance improvements may therefore reflect changes in test difficulty or sample composition. Without a fixed test set, matched random-removal controls, and group-specific removal rates, the experiment cannot support the appendix’s claim that it isolates and quantifies the contribution of color cues.

  3. The central motivation is not sufficiently tested. The paper does not manipulate demographic coverage in the training data or include matched ANN baselines and spiking-parameter ablations. The results establish group disparities in the evaluated SNNs but do not determine the effects of coverage gaps or attribute these disparities to SNN-specific dynamics. Color-related error examples also do not establish texture-based shortcut learning.

  4. The evaluation requires clearer interpretation and more reproducibility details. The RFW metric is effectively a gap in class-wise recall and differs from the fairness settings used on the other datasets. The paper should also specify identity-disjoint splitting, dataset partitions, complete hyperparameters, the number of random seeds, and checkpoint selection criteria, alongside group sample sizes and label distributions.

Questions for authors
  1. Does filtering the training data still improve group disparities on a fixed test set? How does it compare with removing the same number of samples randomly while matching group proportions?
  2. Are the training and test splits identity-disjoint for RFW and DemogPairs? What are the removal rates for each demographic group?
  3. Can the authors provide paired color-intervention results using the same model and images, including the original-image baseline, full confusion matrices, and uncertainty estimates?
  4. What evidence distinguishes SNN-specific effects from general dataset or optimization effects? Which results directly support shortcut learning beyond sensitivity to color transformations?
1 comment
AuthorsOpenReview ↗

Response to Reviewer CCaa

Thank you for the literature corrections and the requests for matched controls and paired color interventions. We address these points and clarify the interpretation of the evaluation metrics.

W1 — Related work and the novelty claim.

We remove the use of Altamirano et al.'s public-health survey as support for an SNN-specific research gap and remove the “first systematic fairness benchmark” claim. Section 1 discusses Perdigão et al.'s CalibraSNN, Nasif et al.'s SNN fraud-detection optimization, and Yang et al.'s Fair-SGNN preprint. We distinguish our facial-image classification benchmark from their fraud-detection and graph settings. The ordinary Bernoulli input ablation is not presented as a reproduction of fairness-aware Bernoulli encoding.

W3 — Matched-control component.

We add UTKFace ANN/SNN ResNet18 controls with equal parameter counts (11,177,538), learned tensor shapes, data, preprocessing, optimizer/search procedure and 150-epoch budgets. Both train from scratch. Five one-factor controls vary timesteps, threshold and encoding; temporal processing and batch-normalization behavior remain different between the ANN and SNN.

ANN/SNN accuracy is 87.92/85.77%, with five-group SP of 16.16/14.69 pp and EO of 10.05/8.92 pp. SNN-minus-ANN accuracy is −2.15 pp [−3.23, −1.06], and fairness contrasts are −1.47 pp [−5.17, 1.56] for SP and −1.13 pp [−6.92, 3.93] for EO. White/Non-White pooling reverses the fairness point rankings, while both intervals again include zero. All five ablations' original-image SP/EO contrast intervals against the SNN anchor include zero; T=2 and Bernoulli reduce accuracy by 2.22 and 2.19 pp. These controls cover one convolutional family (Tables 2, 8 and 10).

These are supplementary controls under a separately specified protocol, rather than repetitions of the historical table entries. All seven formal conditions use seed 0. Validation correct count selects the checkpoint, with the earliest epoch retained in a tie. The 14,049/2,476/4,146 training/validation/evaluation images use exact-byte image components as partition units, with exact-byte overlap excluded between splits. The evaluation pool reuses the benchmark's existing validation images. Paired 95% intervals use 2,000 race-stratified exact-byte-component resamples. They are conditional on the selected models, exclude training-seed and search variability, and have no multiplicity adjustment.

Q3 — Paired RGB/grayscale evaluation on UTKFace.

For each of the seven UTKFace conditions, we evaluate the same selected checkpoint on the same 4,146 images in RGB and grayscale. Grayscale is 0.299R+0.587G+0.114B, replicated across channels after spatial preprocessing and before normalization. Bernoulli validation/evaluation average logits over ten sample/draw-keyed realizations; the RGB/grayscale comparison shares the random streams.

ANN accuracy changes from 87.92% to 82.75%; SNN-anchor accuracy changes from 85.77% to 80.03%. Using unrounded values, the losses are 5.16 and 5.74 pp, and the post hoc SNN-minus-ANN loss contrast is +0.58 pp [−0.70, 1.93]. All seven conditions lose accuracy; group-disparity changes vary, with some fairness-change intervals excluding zero. We provide original-image baselines, all fourteen overall confusion matrices, within-race TN/FP/FN/TP counts and positive-label denominators, and paired intervals (Tables 7, 9, 11–13 and 15–21; Figure 5; Appendices B–E). These paired results describe transformation sensitivity on UTKFace; attributing the changes to texture or skin-color shortcuts would require separate interventions.

Interpretation and metric scope.

The revised abstract, main text and conclusion describe the measured disparities and transformation responses. They no longer present demographic coverage, membrane dynamics or gradient competition as experimentally established causes. We did not manipulate demographic coverage or directly measure the proposed mechanisms.

We correct the scope of the metric description: the cross-architecture gender-classification results use White/Non-White two-group SP/EO. The supplementary UTKFace controls use five recorded race groups (White, Black, Asian, Indian and Other; Female=1) as their primary analysis and additionally report post hoc White/Non-White pooling of the same predictions. The multi-group formula remains valid. EO denotes Equal Opportunity, and the prediction target remains binary gender classification. Section 2.1 now identifies the grouping used for each set of results; the cross-architecture table values are unchanged.

Section 2.2 separately defines the RFW four-class recall gap as class-performance imbalance.

Reviewer_hXRfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work introduces a fairness benchmark for Spiking Neural Networks (SNNs), focusing on demographic disparities and potential reliance on spurious visual features. The benchmark evaluates 12 SNN configurations across four facial image datasets. Additionally, the authors introduce RFW-Cleaned, a dataset variant obtained by removing approximately 17.9% of images identified as grayscale or having abnormal color characteristics.Then the authors compared with the original and cleaned datasets to study the potential influence of color-related shortcuts on demographic disparities. They also analyze group-wise accuracy trajectories during training to examine how disparities emerge and evolve. Experimental results show substantial differences in fairness across SNN architectures and datasets, demonstrating that higher classification accuracy does not necessarily imply better fairness. And they identify asymmetric learning trajectories across demographic groups.

Critical strengths
  1. This work brings up a timely question with broad architectural coverage. Fairness of SNNs on human-centric vision tasks has received little attention. Evaluating 12 configurations, spanning convolutional, residual, and Transformer families with paired TET/non-TET variants, under one protocol gives a reasonably wide empirical snapshot, and the results reveal substantial variation in demographic disparities across architectures and datasets. And the code is released (although I did not run it myself).

  2. Releasing RFW and RFW-Cleaned with an explicit, simple cleaning rule enables follow-up studies of how color artifacts interact with group disparities. The finding that cleaning helps some architectures and hurts others is at least a prompt for further investigation. However, this strength is limited by the weakness discussed below.

Critical weaknesses
  1. Insufficient evidence that the identified fairness issues and mechanisms are specific to SNNs.

The experiments exclusively evaluate SNN architectures but did not include conventional ANN baselines. So it is unclear whether the observed demographic disparities, sensitivity to color-related features, and asymmetric learning trajectories are particular to SNNs or are general phenomena shared by conventional neural networks. Additionally, the authors point to spurious color cues, membrane dynamics, and gradient competition, but they do not directly test whether the models rely on those color cues or examine spike activity and gradients for evidence of the proposed mechanisms. This point is my main concern.

  1. Several experimental designs and results do not support the conclusions drawn from them.

The cleaning step removes 17.9% of RFW images based on color statistics. It changes both what the models train on and which images they are evaluated on. If group accuracy improves or the gap narrows, that could be because the remaining examples are easier or the mix of examples within or across groups has changed. Without controlled interventions on these factors, the cleaning experiment does not convincingly establish the proposed relationship between spurious features and fairness.

RFW also measures something different from the other datasets. In UTKFace, FairFace, and DemogPairs, the model predicts gender and race defines the evaluation groups. In RFW, race serves both roles, so the maximum accuracy gap tells us how much performance varies across classes. It does not have the same interpretation as SP or EO. Although the authors acknowledge this distinction, it weakens the consistency of the proposed fairness benchmark.

Questions for authors
  1. How do the evaluated SNNs compare against conventional ANN architectures with comparable model capacity, training data, and optimization settings? Are the demographic disparities and asymmetric learning trajectories substantially different between SNNs and ANNs? If ANNs show similar disparities, which claims about SNN-specific behavior would remain?
  2. Were QKFormer and Spikformer properly converged on UTKFace and DemogPairs? You may give training curves and hyperparameters, or explain why accuracies are near chance.
  3. How were train and test splits constructed for RFW, which is distributed as a test-only verification set, and for DemogPairs? Are splits identity-disjoint? Leakage of identities would strongly affect both accuracy and per-group gaps.
  4. What fraction of each racial group was removed by cleaning? Can you evaluate models trained on RFW and on RFW-Cleaned against a common test set, to separate training effects from test-population effects?
2 comments
AuthorsOpenReview ↗

Response to Reviewer hXRf

Thank you for asking for ANN comparisons under comparable conditions and a clearer account of what the observations support. Our response focuses on the supplementary UTKFace comparisons and task/metric clarification.

W1 / Q1 — Matched ANN/SNN comparisons.

We compare UTKFace ANN/SNN ResNet18 models with equal parameter counts (11,177,538), learned tensor shapes, data, preprocessing, optimizer/search procedure and 150-epoch budgets. Both train from scratch and select learning rate 0.0005 in validation-only pilots. Five one-factor SNN controls vary timesteps, threshold and encoding. Temporal processing and batch-normalization behavior still differ between the families.

ANN/SNN accuracy is 87.92/85.77%. Five-group SP is 16.16/14.69 pp and EO is 10.05/8.92 pp. SNN-minus-ANN accuracy is −2.15 pp [−3.23, −1.06]; SP and EO contrasts are −1.47 pp [−5.17, 1.56] and −1.13 pp [−6.92, 3.93]. Pooled White/Non-White evaluation changes the SNN-minus-ANN SP/EO contrasts to +0.92 and +0.19 pp, with both intervals again including zero. All five ablations' original-image fairness contrasts against the SNN anchor also include zero. The tables report every ablation, including the accuracy decreases of 2.22 pp for T=2 and 2.19 pp for Bernoulli encoding.

ANN and SNN anchor lose 5.16 and 5.74 pp accuracy on the same grayscale-transformed images. The post hoc SNN-minus-ANN difference in losses is +0.58 pp [−0.70, 1.93]. Thus, both models show transformation sensitivity; these measurements do not establish an SNN-specific mechanism. We frame the contribution as measurement of the evaluated SNN configurations with an ANN reference, and remove assertions that membrane, gradient or demographic-coverage mechanisms were demonstrated.

Figure 1 shows all recorded training/validation accuracy and loss trajectories for the ANN and SNN anchor; Figure 2 shows validation accuracy for all five ablations. These descriptive trajectories summarize the training and validation behavior of the evaluated configurations.

These are supplementary controls under a separately specified protocol, rather than repetitions of the historical table entries. All seven formal conditions use seed 0. Validation correct count selects the checkpoint, with the earliest epoch retained in a tie. The 14,049/2,476/4,146 training/validation/evaluation images use exact-byte image components as partition units, with exact-byte overlap excluded between splits. The evaluation pool reuses the benchmark's existing validation images. Paired 95% intervals use 2,000 race-stratified exact-byte-component resamples. They are conditional on the selected models, exclude training-seed and search variability, and have no multiplicity adjustment.

The controls cover one convolutional family. Locations: Sections 3.2–3.5; Tables 2, 8, 10, 12 and 14; Appendices B–E.

W2 — RFW task interpretation.

Section 2.2 distinguishes the RFW four-class recall gap from gender-classification SP/EO. RFW results describe class-performance imbalance under the implemented classification task, rather than fairness in the original face-verification protocol.

Metric clarification.

We correct the scope of the metric description: the cross-architecture gender-classification results use White/Non-White two-group SP/EO. The supplementary UTKFace controls use five recorded race groups (White, Black, Asian, Indian and Other; Female=1) as their primary analysis and additionally report post hoc White/Non-White pooling of the same predictions. The multi-group formula remains valid. EO denotes Equal Opportunity, and the prediction target remains binary gender classification. Section 2.1 now identifies the grouping used for each set of results; the cross-architecture table values are unchanged.

Thank you for adding the matched ANN baseline and clarifying the scope of the claims. However, my concern remains that the RFW-Cleaned comparison also changes both the training and evaluation sets, so it is unclear whether the reported differences come from filtering the training data or evaluating on different images. The issue still limits my confidence in this work.

Reviewer_qKRpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper evaluates fairness in spiking neural networks on facial classification tasks. The authors compare accuracy and group differences across four datasets. They also remove images with unusual colors and track group accuracy during training. The main contribution is a set of benchmark results, rather than a new method for improving fairness.

Critical strengths
  1. The paper evaluates several SNN configurations on four facial datasets. These results may be useful for future studies.

  2. The authors report group-level results and training curves, in addition to overall accuracy. This helps show differences that overall accuracy alone may hide.

Critical weaknesses
  1. The paper mainly applies existing fairness measures and analysis methods to SNNs. It does not include matched standard CNN or Transformer baselines, or systematically study spike encoding, timesteps, and firing thresholds. The authors should add these controls to show whether the findings are related to the special properties of SNNs.

  2. The other datasets use gender classification and compare performance across racial groups. On RFW, the model predicts race itself, and the paper compares accuracy across classes. These gaps may also reflect differences in class difficulty. Although the authors acknowledge this distinction, they should explain which fairness conclusions this task supports and why they do not use the original face-verification task.

  3. RFW-Cleaned removes images from both sets. Smaller group gaps could partly come from removing difficult evaluation samples, rather than reducing shortcut learning. The authors should compare training conditions on a fixed test set and report the number and percentage of removed images for each group.

  4. Appendix C does not clearly report the split ratios, whether splits are identity-disjoint, or the number of repeated runs. The tables give means and standard deviations, but these details are needed to assess the results. Please clarify whether images of the same person can appear in different splits.

  5. The main text presents color shortcuts and gradient effects as possible explanations, while parts of the abstract, conclusion, and appendix use stronger wording. The authors should clearly separate observations from possible explanations. The face examples in Figures 2 and 4 are also too small and lack clear sample-level prediction labels. Figure 5 has crowded curves and small text, making the main comparisons difficult to check.

Questions for authors

Please refer to the Weaknesses part.

2 comments
AuthorsOpenReview ↗

Response to Reviewer qKRp

Thank you for the requests for comparable controls and clearer interpretation. We provide supplementary UTKFace comparisons and clarify the metric definitions, RFW task scope, and presentation.

W1 — Matched CNN control and spiking ablations.

We compare ANN and SNN ResNet18 on UTKFace with 11,177,538 parameters each, matching learned tensor shapes, data, preprocessing, optimizer/search procedure and a 150-epoch budget. Both select learning rate 0.0005 in validation-only pilots and train from scratch. The SNN anchor uses T=4, threshold 1 and direct encoding. Five one-factor ablations change T to 2 or 8, threshold to 0.5 or 1.5, or encoding to Bernoulli RGB. ReLU/LIF processing, temporal aggregation and batch-normalization behavior remain different.

ANN/SNN accuracy is 87.92/85.77%; five-group SP is 16.16/14.69 pp and EO is 10.05/8.92 pp. SNN-minus-ANN accuracy is −2.15 pp [−3.23, −1.06], while SP and EO contrasts are −1.47 pp [−5.17, 1.56] and −1.13 pp [−6.92, 3.93]. White/Non-White pooling changes the SP/EO point contrasts to +0.92 and +0.19 pp; their intervals also include zero. All five ablations' original-image SP/EO contrasts against the anchor include zero under both groupings. T=2 and Bernoulli reduce accuracy by 2.22 and 2.19 pp. All conditions are reported in Tables 2, 8 and 10. The comparison covers one convolutional family, without a matched ANN Transformer.

These are supplementary controls under a separately specified protocol, rather than repetitions of the historical table entries. All seven formal conditions use seed 0. Validation correct count selects the checkpoint, with the earliest epoch retained in a tie. The 14,049/2,476/4,146 training/validation/evaluation images use exact-byte image components as partition units, with exact-byte overlap excluded between splits. The evaluation pool reuses the benchmark's existing validation images. Paired 95% intervals use 2,000 race-stratified exact-byte-component resamples. They are conditional on the selected models, exclude training-seed and search variability, and have no multiplicity adjustment.

Locations in the revised draft: Sections 3.2–3.3 and Appendices B–D.

Metric clarification.

We correct the scope of the metric description: the cross-architecture gender-classification results use White/Non-White two-group SP/EO. The supplementary UTKFace controls use five recorded race groups (White, Black, Asian, Indian and Other; Female=1) as their primary analysis and additionally report post hoc White/Non-White pooling of the same predictions. The multi-group formula remains valid. EO denotes Equal Opportunity, and the prediction target remains binary gender classification. Section 2.1 now identifies the grouping used for each set of results; the cross-architecture table values are unchanged.

W2 — Interpretation of the RFW task.

Our RFW implementation predicts the four recorded race categories. Its maximum class-recall gap describes class-performance imbalance and may also reflect differences in class difficulty. It is distinct from gender-classification SP/EO with race as a separate sensitive attribute. We clarify this scope in Section 2.2. The experiments use a four-class classifier rather than a verification protocol, so the reported results are confined to that classification task. We do not present this task as a substitute for face verification.

W5 — Interpretation and training-figure presentation.

We align the abstract, main text and conclusion around measured group disparities and transformation sensitivity. Membrane dynamics, gradient competition and texture shortcuts are not presented as experimentally established mechanisms. The two components of the original Figure 5 are presented separately as Figures 6–7, with independent legends, T/F definitions and broken-axis explanations. Epoch 0 denotes evaluation after the first complete training epoch; early values are tabulated separately.

Thank you for the response and the additional experiments. The matched ANN comparison and the clearer discussion of the RFW task address some of my concerns. However, the new results do not provide clear evidence that the fairness findings are specific to SNNs. The RFW-Cleaned results also remain difficult to interpret because both the training and evaluation sets change, so the effects of cleaning cannot be separated from changes in evaluation difficulty. I appreciate the revisions, but these concerns still limit the strength of the paper’s conclusions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.