- Rating
- 2(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper presents a benchmark for evaluating fairness in SNNs on facial image datasets. It evaluates gender classification with race as the sensitive attribute on three datasets and examines group disparities in race-classification accuracy on RFW. The authors also construct RFW-Cleaned to compare results before and after color-based filtering and analyze group-wise learning trajectories. The results show that overall accuracy does not adequately capture group disparities and that the effects of filtering depend on the model configuration.
Critical strengths
- The paper addresses the relatively underexplored topic of SNN fairness and covers multiple model families and datasets, providing a useful benchmarking reference.
- Reporting accuracy, SP, and EO together demonstrates that different metrics can produce different model rankings.
- The analysis extends beyond final performance to group-wise learning trajectories and acknowledges that filtering does not benefit every configuration.
Critical weaknesses
-
The citation supporting the research-gap claim appears mismatched. Altamirano et al. (2025) reviews algorithmic bias in public-health machine learning and does not appear to substantiate the claim that fairness is largely overlooked in SNNs. Moreover, recent work has already examined fairness in SNNs, including Perdigão et al. (2025), CalibraSNN: Fair and Calibrated Convolutional Spiking Neural Network for High-Stakes Industry Applications; Nasif et al. (2026), Reinforcement-guided hyper-heuristic hyperparameter optimization for fair and explainable spiking neural network-based financial fraud detection; and Yang et al. (2026), Towards Fair Spiking Graph Neural Networks via Debiased Representation Learning and Fairness-Aware Bernoulli Encoding. The authors should provide an SNN-focused literature review, qualify the “largely overlooked” claim, and clarify how their benchmark differs from existing work, particularly in relation to the claimed novelty as the “first systematic fairness benchmark.”
-
The filtering experiment does not isolate the effects of color. Comparing the original and cleaned datasets changes both the training and evaluation samples. Performance improvements may therefore reflect changes in test difficulty or sample composition. Without a fixed test set, matched random-removal controls, and group-specific removal rates, the experiment cannot support the appendix’s claim that it isolates and quantifies the contribution of color cues.
-
The central motivation is not sufficiently tested. The paper does not manipulate demographic coverage in the training data or include matched ANN baselines and spiking-parameter ablations. The results establish group disparities in the evaluated SNNs but do not determine the effects of coverage gaps or attribute these disparities to SNN-specific dynamics. Color-related error examples also do not establish texture-based shortcut learning.
-
The evaluation requires clearer interpretation and more reproducibility details. The RFW metric is effectively a gap in class-wise recall and differs from the fairness settings used on the other datasets. The paper should also specify identity-disjoint splitting, dataset partitions, complete hyperparameters, the number of random seeds, and checkpoint selection criteria, alongside group sample sizes and label distributions.
Questions for authors
- Does filtering the training data still improve group disparities on a fixed test set? How does it compare with removing the same number of samples randomly while matching group proportions?
- Are the training and test splits identity-disjoint for RFW and DemogPairs? What are the removal rates for each demographic group?
- Can the authors provide paired color-intervention results using the same model and images, including the original-image baseline, full confusion matrices, and uncertainty estimates?
- What evidence distinguishes SNN-specific effects from general dataset or optimization effects? Which results directly support shortcut learning beyond sensitivity to color transformations?
Response to Reviewer CCaa
Thank you for the literature corrections and the requests for matched controls and paired color interventions. We address these points and clarify the interpretation of the evaluation metrics.
W1 — Related work and the novelty claim.
We remove the use of Altamirano et al.'s public-health survey as support for an SNN-specific research gap and remove the “first systematic fairness benchmark” claim. Section 1 discusses Perdigão et al.'s CalibraSNN, Nasif et al.'s SNN fraud-detection optimization, and Yang et al.'s Fair-SGNN preprint. We distinguish our facial-image classification benchmark from their fraud-detection and graph settings. The ordinary Bernoulli input ablation is not presented as a reproduction of fairness-aware Bernoulli encoding.
W3 — Matched-control component.
We add UTKFace ANN/SNN ResNet18 controls with equal parameter counts (11,177,538), learned tensor shapes, data, preprocessing, optimizer/search procedure and 150-epoch budgets. Both train from scratch. Five one-factor controls vary timesteps, threshold and encoding; temporal processing and batch-normalization behavior remain different between the ANN and SNN.
ANN/SNN accuracy is 87.92/85.77%, with five-group SP of 16.16/14.69 pp and EO of 10.05/8.92 pp. SNN-minus-ANN accuracy is −2.15 pp [−3.23, −1.06], and fairness contrasts are −1.47 pp [−5.17, 1.56] for SP and −1.13 pp [−6.92, 3.93] for EO. White/Non-White pooling reverses the fairness point rankings, while both intervals again include zero. All five ablations' original-image SP/EO contrast intervals against the SNN anchor include zero; T=2 and Bernoulli reduce accuracy by 2.22 and 2.19 pp. These controls cover one convolutional family (Tables 2, 8 and 10).
These are supplementary controls under a separately specified protocol, rather than repetitions of the historical table entries. All seven formal conditions use seed 0. Validation correct count selects the checkpoint, with the earliest epoch retained in a tie. The 14,049/2,476/4,146 training/validation/evaluation images use exact-byte image components as partition units, with exact-byte overlap excluded between splits. The evaluation pool reuses the benchmark's existing validation images. Paired 95% intervals use 2,000 race-stratified exact-byte-component resamples. They are conditional on the selected models, exclude training-seed and search variability, and have no multiplicity adjustment.
Q3 — Paired RGB/grayscale evaluation on UTKFace.
For each of the seven UTKFace conditions, we evaluate the same selected checkpoint on the same 4,146 images in RGB and grayscale. Grayscale is 0.299R+0.587G+0.114B, replicated across channels after spatial preprocessing and before normalization. Bernoulli validation/evaluation average logits over ten sample/draw-keyed realizations; the RGB/grayscale comparison shares the random streams.
ANN accuracy changes from 87.92% to 82.75%; SNN-anchor accuracy changes from 85.77% to 80.03%. Using unrounded values, the losses are 5.16 and 5.74 pp, and the post hoc SNN-minus-ANN loss contrast is +0.58 pp [−0.70, 1.93]. All seven conditions lose accuracy; group-disparity changes vary, with some fairness-change intervals excluding zero. We provide original-image baselines, all fourteen overall confusion matrices, within-race TN/FP/FN/TP counts and positive-label denominators, and paired intervals (Tables 7, 9, 11–13 and 15–21; Figure 5; Appendices B–E). These paired results describe transformation sensitivity on UTKFace; attributing the changes to texture or skin-color shortcuts would require separate interventions.
Interpretation and metric scope.
The revised abstract, main text and conclusion describe the measured disparities and transformation responses. They no longer present demographic coverage, membrane dynamics or gradient competition as experimentally established causes. We did not manipulate demographic coverage or directly measure the proposed mechanisms.
We correct the scope of the metric description: the cross-architecture gender-classification results use White/Non-White two-group SP/EO. The supplementary UTKFace controls use five recorded race groups (White, Black, Asian, Indian and Other; Female=1) as their primary analysis and additionally report post hoc White/Non-White pooling of the same predictions. The multi-group formula remains valid. EO denotes Equal Opportunity, and the prediction target remains binary gender classification. Section 2.1 now identifies the grouping used for each set of results; the cross-architecture table values are unchanged.
Section 2.2 separately defines the RFW four-class recall gap as class-performance imbalance.