- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes SDSR-HM for evaluating fairness in federated ReID under domain skew. It divides each domain's federated performance by a single-domain reference obtained through independent training under matched experimental settings. These achievement ratios are capped at 1 and aggregated using the harmonic mean, separately for mAP and Rank-1. Capping prevents above-reference gains on one domain from offsetting shortfalls elsewhere, while harmonic aggregation gives domains with lower ratios greater influence on the score. Comparisons on four ReID datasets with ResNet50 and ViT, together with simulated reductions in domain scores, show that SDSR-HM distinguishes methods with similar raw-score standard deviations but different achievement ratios.
Critical strengths
-
The paper identifies an important limitation of evaluating fairness solely through performance disparities: small disparities can coexist with poor overall performance or performance far below single-domain references.
-
The computation and the roles of single-domain references, ratio capping, and harmonic aggregation are clearly explained, making score differences between methods easier to interpret.
Critical weaknesses
-
The authors explain that SDSR-HM is more sensitive to lower achievement ratios, but the experiments provide limited evidence of how this difference helps evaluate competing methods. Q and SDSR-HM yield the same method rankings in Table 2, and no separate comparison without ratio capping is provided.
-
Reference models are trained using one random seed and a fixed number of epochs, without testing how changes in reference scores affect method rankings. Federated training budgets and adaptations to the unified source-domain protocol are unclear, making it difficult to assess whether comparison conditions are consistent.
Questions for authors
-
Compared with study [1], what are the substantive differences in this paper's definition of fairness, construction of reference models, and aggregation of per-domain results?
-
On what basis were the inferred degradation factors used in Figure 3b for the identity-skew, data-imbalance, and camera-shift settings determined?
-
Could the authors clarify the application scenarios in which assigning equal weight to each client is appropriate? When service volume differs across domains, would weighting by factors such as query volume significantly change method rankings?
References: [1] Divi S, Lin Y S, Farrukh H, et al. New metrics to evaluate the performance and fairness of personalized federated learning[J]. arXiv preprint arXiv:2107.13173, 2021.
3 comments
Response to Comment 2: Determination of Heterogeneity Degradation Factors in Figure 3(b)
We thank the reviewer for raising this important question regarding the determination of the heterogeneity factors used in Figure 3(b).
We clarify that these factors are grounded in the actual data heterogeneity characteristics of the ReID datasets and their corresponding experimental configurations, rather than being arbitrarily assigned performance degradation coefficients. Specifically, we construct three types of heterogeneous client settings based on the actual identity labels, training sample distributions, and camera annotations. The underlying rationale for each setting is detailed below.
(1) Identity Skew.
The identity-skew setting is determined by the distribution of identity labels across clients. Specifically, we examine identity allocation and identity coverage among clients and construct heterogeneous identity distributions with different levels of identity concentration. Such heterogeneity reduces the diversity and uniformity of identity information available to different clients, potentially affecting identity representation learning and cross-domain retrieval performance.
(2) Data Imbalance.
The data-imbalance setting is determined by the actual number of training images available to each client. We construct different levels of imbalance by varying the relative amounts of training data across clients while retaining the underlying ReID data and identity annotations. These differences in local data availability result in unequal training support and optimization conditions across domains.
(3) Camera Shift.
The camera-shift setting is based on the actual camera annotations and camera-dependent image distributions. Different cameras capture pedestrians under varying viewpoints, illumination conditions, backgrounds, and visual appearances. We construct camera-dependent heterogeneous settings based on these real camera-specific data characteristics to investigate their influence on cross-domain retrieval performance.
(4) Experimental Results and Metric Responses.
For each heterogeneity setting, the corresponding federated model is evaluated on the four ReID domains using the same evaluation protocol and fixed domain-specific supervised references. Therefore, the heterogeneity configurations are determined by the actual data distributions, while the reported SDSR-HM scores are calculated from the resulting domain-level experimental performance.
As shown in Figure 3(b), SDSR-HM decreases from 95.0 in the original setting to 59.9 under identity skew, 56.0 under data imbalance, and 51.3 under camera shift. These results demonstrate that SDSR-HM responds to different sources of data heterogeneity and captures the resulting reduction in reference-relative domain achievement.
Meanwhile, the corresponding transformed STD-based scores are 70.9, 72.5, and 74.6, respectively. Their different responses further illustrate that raw performance dispersion and reference-relative domain achievement characterize different aspects of cross-domain fairness.
Overall, the three heterogeneity factors are determined based on actual identity allocation, client-level data availability, and camera-dependent data distributions in the ReID experiments. This design enables us to systematically examine the sensitivity of SDSR-HM to different practically relevant sources of domain heterogeneity, while highlighting its distinction from conventional performance-dispersion-based evaluation.
Response to Comment 3: Client Weighting and Query-Volume Sensitivity
We thank the reviewer for this insightful question.
(1) When is equal client weighting appropriate?
Equal client weighting is appropriate when clients represent independent service units with comparable priorities, such as geographically distributed surveillance networks, public safety camera systems, and multi-institutional ReID applications. In these scenarios, smaller clients should not be overlooked simply because they receive fewer retrieval requests. Equal weighting therefore prioritizes equitable recognition quality across clients rather than maximizing aggregate performance on high-volume domains.
(2) Sensitivity to query-volume-based weighting.
To investigate the influence of different service volumes, we conduct a post-hoc sensitivity analysis using existing domain-level results while keeping model predictions and reference models unchanged. Three representative workload distributions are considered:
Table 1. Domain-weighting configurations.
+-------------+----------+----------+----------+----------+
| Domain | Equal | Scenario A| Scenario B| Scenario C|
+-------------+----------+----------+----------+----------+
| Market-1501 | 25% | 40% | 10% | 10% |
| MSMT17 | 25% | 30% | 20% | 50% |
| CUHK03 | 25% | 20% | 30% | 20% |
| CUHK02 | 25% | 10% | 40% | 20% |
+-------------+----------+----------+----------+----------+
Table 2. Ranking sensitivity under different weighting configurations.
+----------------------+------------+------------+------------+
| Evaluation Metric | Scenario A | Scenario B | Scenario C |
+----------------------+------------+------------+------------+
| Spearman (mAP) | 0.995 | 0.962 | 0.967 |
| Spearman (Rank-1) | 0.929 | 1.000 | 0.885 |
+----------------------+------------+------------+------------+
The results indicate that query-volume weighting can change the relative rankings of some intermediate methods, particularly when certain domains receive substantially higher weights. Nevertheless, the overall ranking structure remains relatively consistent across the three scenarios.
These findings demonstrate that equal weighting is suitable for fairness-oriented evaluation across independent clients, whereas query-volume weighting is more appropriate for evaluating overall system utility under unequal service demands. We therefore consider the two strategies complementary rather than interchangeable.
Response to Comment 1: Differences from Divi et al. [1]
We thank the reviewer for pointing out the related work by Divi et al. [1]. We agree that the distinctions between their fairness evaluation framework and our proposed SDSR-HM metric deserve further clarification. Although both studies investigate fairness in federated learning, they address fundamentally different fairness objectives, reference constructions, and aggregation strategies. We clarify these differences from three perspectives.
(1) Differences in Fairness Definitions.
Divi et al. [1] focus on personalized federated learning and evaluate whether individual clients receive equitable performance improvements from personalization. Their framework considers the performance differences between personalized models, global models, and local models. They further employ statistical measures, including Average Variance, Cosine Similarity, Entropy, and Jain's Index, to characterize the distribution of personalization gains across clients.
In contrast, our work focuses on cross-domain performance fairness in federated person re-identification (ReID). Our objective is to evaluate how effectively a federated model preserves the achievable retrieval capability of each heterogeneous domain. This distinction is particularly important in ReID, where different datasets exhibit substantially different retrieval difficulties. Consequently, similar performance dispersion across domains does not necessarily indicate comparable levels of fairness.
(2) Differences in Reference-Model Construction.
Divi et al. [1] use local and global model performance as reference points to evaluate personalized improvements. In contrast, our framework independently constructs a Single-Domain Supervised Reference (SDSR) for each ReID domain, including Market-1501, MSMT17, CUHK03, and CUHK02.
Each reference model is trained independently using its corresponding domain-specific data, without federated communication or cross-domain aggregation. We maintain the same backbone architecture and a consistent ReID training protocol as those used in the federated experiments. Furthermore, separate reference models are constructed for ResNet50 and ViT to ensure architectural consistency.
Importantly, these references are not intended to represent historical state-of-the-art performance or absolute theoretical upper bounds. Instead, they provide domain-specific supervised performance anchors, allowing us to assess how much of each domain's reference-level retrieval capability is retained by federated learning.
(3) Differences in Aggregation Strategies.
Divi et al. [1] primarily characterize the distribution of client-level personalization improvements through multiple statistical fairness measures. In contrast, our approach first normalizes each domain's retrieval performance using its corresponding SDSR reference and then aggregates the resulting domain-level achievement scores using the harmonic mean.
This design makes the overall score more sensitive to domains with relatively low achievement. In particular, a method cannot obtain a favorable fairness score simply because its performance is similarly low across all domains, which is a limitation of relying solely on performance dispersion.
Overall, our contribution is not a direct adaptation of existing personalized federated learning fairness metrics. Rather, we introduce a domain-reference-based fairness evaluation framework specifically designed to address heterogeneous retrieval difficulties in federated ReID. By incorporating domain-specific supervised references and harmonic aggregation, SDSR-HM provides a complementary perspective for evaluating cross-domain fairness beyond conventional client-level personalization fairness measures.