acceptodds
Under review as a conference paper at ICLR 2027

Rethinking Fairness Evaluation in Federated ReID: Beyond Performance Variance under Domain Skew

Abstract

Fair federated ReID aims to achieve equitable identity retrieval across decen- tralized and heterogeneous ReID domains. However, when fairness is evaluated solely by performance parity among domains, smaller retrieval performance vari- ance is regarded as stronger fairness. Used alone, this criterion overlooks a fun- damental property of ReID scenarios: different domains inherently possess dif- ferent retrieval difficulties caused by variations in cameras, viewpoints, back- grounds, illumination, and identity distributions. Consequently, equal raw perfor- mance does not necessarily imply equal reference-relative attainment, while low variance can coexist with uniformly poor performance. In this work, we rethink the fairness objective of fair federated ReID and argue that fair achievement should be evaluated relative to each domain’s protocol-matched single-domain reference rather than raw performance disparity alone. To this end, we introduce Single- Domain Supervised Reference (SDSR), a reference calibration scheme that cali- brates federated ReID performance using protocol-matched single-domain refer- ence performance. Based on normalized domain achievement, we further propose SDSR-HM, which adopts harmonic aggregation to emphasize balanced reference- relative attainment across heterogeneous and challenging ReID domains. Exten- sive experiments demonstrate that raw dispersion alone can produce mislead- ing fair-achievement rankings and fail to distinguish high attainment from uni- form under-attainment. Our results reveal the necessity of considering reference- relative attainment in fair federated ReID and provide a more informative criterion for measuring fair achievement under domain skew.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Federated ReID, Fair Federated

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_3d9kOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes SDSR-HM for evaluating fairness in federated ReID under domain skew. It divides each domain's federated performance by a single-domain reference obtained through independent training under matched experimental settings. These achievement ratios are capped at 1 and aggregated using the harmonic mean, separately for mAP and Rank-1. Capping prevents above-reference gains on one domain from offsetting shortfalls elsewhere, while harmonic aggregation gives domains with lower ratios greater influence on the score. Comparisons on four ReID datasets with ResNet50 and ViT, together with simulated reductions in domain scores, show that SDSR-HM distinguishes methods with similar raw-score standard deviations but different achievement ratios.

Critical strengths
  1. The paper identifies an important limitation of evaluating fairness solely through performance disparities: small disparities can coexist with poor overall performance or performance far below single-domain references.

  2. The computation and the roles of single-domain references, ratio capping, and harmonic aggregation are clearly explained, making score differences between methods easier to interpret.

Critical weaknesses
  1. The authors explain that SDSR-HM is more sensitive to lower achievement ratios, but the experiments provide limited evidence of how this difference helps evaluate competing methods. Q and SDSR-HM yield the same method rankings in Table 2, and no separate comparison without ratio capping is provided.

  2. Reference models are trained using one random seed and a fixed number of epochs, without testing how changes in reference scores affect method rankings. Federated training budgets and adaptations to the unified source-domain protocol are unclear, making it difficult to assess whether comparison conditions are consistent.

Questions for authors
  1. Compared with study [1], what are the substantive differences in this paper's definition of fairness, construction of reference models, and aggregation of per-domain results?

  2. On what basis were the inferred degradation factors used in Figure 3b for the identity-skew, data-imbalance, and camera-shift settings determined?

  3. Could the authors clarify the application scenarios in which assigning equal weight to each client is appropriate? When service volume differs across domains, would weighting by factors such as query volume significantly change method rankings?

References: [1] Divi S, Lin Y S, Farrukh H, et al. New metrics to evaluate the performance and fairness of personalized federated learning[J]. arXiv preprint arXiv:2107.13173, 2021.

3 comments
AuthorsOpenReview ↗

Response to Comment 1: Differences from Divi et al. [1]

We thank the reviewer for pointing out the related work by Divi et al. [1]. We agree that the distinctions between their fairness evaluation framework and our proposed SDSR-HM metric deserve further clarification. Although both studies investigate fairness in federated learning, they address fundamentally different fairness objectives, reference constructions, and aggregation strategies. We clarify these differences from three perspectives.

(1) Differences in Fairness Definitions.

Divi et al. [1] focus on personalized federated learning and evaluate whether individual clients receive equitable performance improvements from personalization. Their framework considers the performance differences between personalized models, global models, and local models. They further employ statistical measures, including Average Variance, Cosine Similarity, Entropy, and Jain's Index, to characterize the distribution of personalization gains across clients.

In contrast, our work focuses on cross-domain performance fairness in federated person re-identification (ReID). Our objective is to evaluate how effectively a federated model preserves the achievable retrieval capability of each heterogeneous domain. This distinction is particularly important in ReID, where different datasets exhibit substantially different retrieval difficulties. Consequently, similar performance dispersion across domains does not necessarily indicate comparable levels of fairness.

(2) Differences in Reference-Model Construction.

Divi et al. [1] use local and global model performance as reference points to evaluate personalized improvements. In contrast, our framework independently constructs a Single-Domain Supervised Reference (SDSR) for each ReID domain, including Market-1501, MSMT17, CUHK03, and CUHK02.

Each reference model is trained independently using its corresponding domain-specific data, without federated communication or cross-domain aggregation. We maintain the same backbone architecture and a consistent ReID training protocol as those used in the federated experiments. Furthermore, separate reference models are constructed for ResNet50 and ViT to ensure architectural consistency.

Importantly, these references are not intended to represent historical state-of-the-art performance or absolute theoretical upper bounds. Instead, they provide domain-specific supervised performance anchors, allowing us to assess how much of each domain's reference-level retrieval capability is retained by federated learning.

(3) Differences in Aggregation Strategies.

Divi et al. [1] primarily characterize the distribution of client-level personalization improvements through multiple statistical fairness measures. In contrast, our approach first normalizes each domain's retrieval performance using its corresponding SDSR reference and then aggregates the resulting domain-level achievement scores using the harmonic mean.

This design makes the overall score more sensitive to domains with relatively low achievement. In particular, a method cannot obtain a favorable fairness score simply because its performance is similarly low across all domains, which is a limitation of relying solely on performance dispersion.

Overall, our contribution is not a direct adaptation of existing personalized federated learning fairness metrics. Rather, we introduce a domain-reference-based fairness evaluation framework specifically designed to address heterogeneous retrieval difficulties in federated ReID. By incorporating domain-specific supervised references and harmonic aggregation, SDSR-HM provides a complementary perspective for evaluating cross-domain fairness beyond conventional client-level personalization fairness measures.

AuthorsOpenReview ↗

Response to Comment 2: Determination of Heterogeneity Degradation Factors in Figure 3(b)

We thank the reviewer for raising this important question regarding the determination of the heterogeneity factors used in Figure 3(b).

We clarify that these factors are grounded in the actual data heterogeneity characteristics of the ReID datasets and their corresponding experimental configurations, rather than being arbitrarily assigned performance degradation coefficients. Specifically, we construct three types of heterogeneous client settings based on the actual identity labels, training sample distributions, and camera annotations. The underlying rationale for each setting is detailed below.

(1) Identity Skew.

The identity-skew setting is determined by the distribution of identity labels across clients. Specifically, we examine identity allocation and identity coverage among clients and construct heterogeneous identity distributions with different levels of identity concentration. Such heterogeneity reduces the diversity and uniformity of identity information available to different clients, potentially affecting identity representation learning and cross-domain retrieval performance.

(2) Data Imbalance.

The data-imbalance setting is determined by the actual number of training images available to each client. We construct different levels of imbalance by varying the relative amounts of training data across clients while retaining the underlying ReID data and identity annotations. These differences in local data availability result in unequal training support and optimization conditions across domains.

(3) Camera Shift.

The camera-shift setting is based on the actual camera annotations and camera-dependent image distributions. Different cameras capture pedestrians under varying viewpoints, illumination conditions, backgrounds, and visual appearances. We construct camera-dependent heterogeneous settings based on these real camera-specific data characteristics to investigate their influence on cross-domain retrieval performance.

(4) Experimental Results and Metric Responses.

For each heterogeneity setting, the corresponding federated model is evaluated on the four ReID domains using the same evaluation protocol and fixed domain-specific supervised references. Therefore, the heterogeneity configurations are determined by the actual data distributions, while the reported SDSR-HM scores are calculated from the resulting domain-level experimental performance.

As shown in Figure 3(b), SDSR-HM decreases from 95.0 in the original setting to 59.9 under identity skew, 56.0 under data imbalance, and 51.3 under camera shift. These results demonstrate that SDSR-HM responds to different sources of data heterogeneity and captures the resulting reduction in reference-relative domain achievement.

Meanwhile, the corresponding transformed STD-based scores are 70.9, 72.5, and 74.6, respectively. Their different responses further illustrate that raw performance dispersion and reference-relative domain achievement characterize different aspects of cross-domain fairness.

Overall, the three heterogeneity factors are determined based on actual identity allocation, client-level data availability, and camera-dependent data distributions in the ReID experiments. This design enables us to systematically examine the sensitivity of SDSR-HM to different practically relevant sources of domain heterogeneity, while highlighting its distinction from conventional performance-dispersion-based evaluation.

AuthorsOpenReview ↗

Response to Comment 3: Client Weighting and Query-Volume Sensitivity

We thank the reviewer for this insightful question.

(1) When is equal client weighting appropriate?

Equal client weighting is appropriate when clients represent independent service units with comparable priorities, such as geographically distributed surveillance networks, public safety camera systems, and multi-institutional ReID applications. In these scenarios, smaller clients should not be overlooked simply because they receive fewer retrieval requests. Equal weighting therefore prioritizes equitable recognition quality across clients rather than maximizing aggregate performance on high-volume domains.

(2) Sensitivity to query-volume-based weighting.

To investigate the influence of different service volumes, we conduct a post-hoc sensitivity analysis using existing domain-level results while keeping model predictions and reference models unchanged. Three representative workload distributions are considered:

Table 1. Domain-weighting configurations.

+-------------+----------+----------+----------+----------+
| Domain      | Equal    | Scenario A| Scenario B| Scenario C|
+-------------+----------+----------+----------+----------+
| Market-1501 |   25%    |   40%    |   10%    |   10%    |
| MSMT17      |   25%    |   30%    |   20%    |   50%    |
| CUHK03      |   25%    |   20%    |   30%    |   20%    |
| CUHK02      |   25%    |   10%    |   40%    |   20%    |
+-------------+----------+----------+----------+----------+

Table 2. Ranking sensitivity under different weighting configurations.

+----------------------+------------+------------+------------+
| Evaluation Metric    | Scenario A | Scenario B | Scenario C |
+----------------------+------------+------------+------------+
| Spearman (mAP)       |   0.995    |   0.962    |   0.967    |
| Spearman (Rank-1)    |   0.929    |   1.000    |   0.885    |
+----------------------+------------+------------+------------+

The results indicate that query-volume weighting can change the relative rankings of some intermediate methods, particularly when certain domains receive substantially higher weights. Nevertheless, the overall ranking structure remains relatively consistent across the three scenarios.

These findings demonstrate that equal weighting is suitable for fairness-oriented evaluation across independent clients, whereas query-volume weighting is more appropriate for evaluating overall system utility under unequal service demands. We therefore consider the two strategies complementary rather than interchangeable.

Reviewer_eCWxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper argues that low variation in retrieval performance across federated ReID domains does not necessarily indicate fair achievement: uniformly poor models may have low variance, while domains can differ inherently in difficulty. It proposes SDSR-HM, which compares each domain’s federated performance with a protocol-matched single-domain reference, caps the resulting ratio at one, and aggregates the ratios using a harmonic mean to emphasize domains with lower attainment. Experiments on four ReID datasets with ResNet50 and ViT show that methods with similar or lower raw performance dispersion can have substantially different reference-relative achievement.

Critical strengths
  1. The paper shows why raw cross-domain performance dispersion can be misleading when ReID domains differ in difficulty. Comparing each domain with a protocol-matched single-domain reference gives “fair achievement” a clearer operational meaning.
  2. The capped achievement ratio prevents gains beyond one domain’s reference from compensating for another domain’s shortfall. Harmonic aggregation makes low-achievement domains more influential. The paper explains these properties analytically and illustrates them across two backbones.
Critical weaknesses
  1. Several examples show that STD alone is misleading, but the experiments already report mean retrieval performance alongside STD. The paper does not convincingly establish when SDSR-HM leads to a better assessment than considering those two measures together, or than simpler reference-relative summaries.
  2. The single-domain references use one reported seed and a fixed final checkpoint. The paper does not quantify how reference variability affects SDSR-HM or method rankings. Figure 3 uses constructed degradations, so it demonstrates metric sensitivity rather than robustness under independently trained heterogeneous settings.
Questions for authors
  1. How stable are the single-domain references, SDSR-HM scores, and key method rankings across random seeds, checkpoint choices, and reasonable training-budget changes?
  2. Can you identify a real experimental comparison where mean performance and STD together remain inconclusive, but SDSR-HM yields a different and demonstrably more appropriate judgment?
  3. What exact training budgets, model-selection rules, and adaptation choices were used for the baselines in Table 1? How can readers verify that these choices make the comparison with FedHRA fair and reproducible?
4 comments
AuthorsOpenReview ↗

Response to Comment 1: Stability of Single-Domain References

We thank the reviewer for raising this important concern regarding the reliability of the single-domain references. To investigate reference sensitivity, we examine three aspects: random seed, checkpoint selection, and training budget. For each domain and backbone, we independently train reference models using different configurations while keeping the federated models fixed, thereby isolating the uncertainty introduced by reference construction.

Table 1. Sensitivity analysis of single-domain references.

+------------------+----------------------+----------------------+----------------------+
| Variation Factor | Experimental Setup  | SDSR-HM Variation    | Ranking Consistency  |
+------------------+----------------------+----------------------+----------------------+
| Random Seed      | 40, 2026, 3888       | Std: 0.4-0.6 pp      | Spearman: 0.95-0.96  |
|                  |                      |                      | Kendall: 0.90-0.92   |
+------------------+----------------------+----------------------+----------------------+
| Checkpoint       | Epoch 60, 70, 80     |                      |                      |
| ResNet50-mAP     |                      | 0.73 pp              |                      |
| ResNet50-Rank-1  |                      | 0.78 pp              |                      |
| ViT-mAP          |                      | 1.08 pp              |                      |
| ViT-Rank-1       |                      | 0.50 pp              |                      |
+------------------+----------------------+----------------------+----------------------+
| Training Budget  | 60, 80, 100 epochs   | Relatively small     |                      |
|                  |                      | variations           |                      |
+------------------+----------------------+----------------------+----------------------+

First, under different random seeds, the standard deviations of SDSR-HM remain within 0.4-0.6 percentage points across the evaluated backbone-metric combinations. The corresponding method rankings also remain highly consistent, with Spearman correlations of 0.95-0.96 and Kendall correlations of 0.90-0.92.

Second, checkpoint sensitivity is evaluated using reference models from epochs 60, 70, and 80. The SDSR-HM variations are 0.73 and 0.78 percentage points for ResNet50 mAP and Rank-1, respectively, and 1.08 and 0.50 percentage points for ViT. These results indicate that reasonable checkpoint variations have a limited influence on the resulting fairness scores.

Third, we evaluate reference models trained for 60, 80, and 100 epochs. The resulting SDSR-HM scores exhibit relatively small variations, suggesting that moderate changes in training budgets do not substantially affect the overall evaluation.

Overall, these results demonstrate that SDSR-HM scores and method rankings remain reasonably stable against variations in reference initialization, checkpoint selection, and training duration. We will incorporate these sensitivity analyses and further clarify the reference construction protocol in the revised manuscript.

AuthorsOpenReview ↗

Response to Comment 2: Empirical Evidence Beyond Mean and Standard Deviation

We thank the reviewer for requesting a concrete empirical example demonstrating the practical value of SDSR-HM beyond conventional mean and standard deviation statistics.

We identify two illustrative comparisons from Table 1 under the ResNet50 backbone.

First, comparison between FedReID and FedSupWA.

FedReID achieves an average mAP of 51.8% with a standard deviation of 18.4%, while FedSupWA obtains 50.1% and 19.7%, respectively. Their raw statistics are relatively close, with differences of only 1.7 and 1.3 percentage points. However, their SDSR-HM scores differ more substantially, indicating different levels of reference-relative domain achievement.

Table 1. Comparison of conventional statistics and SDSR-HM.

+------------+----------+----------+----------+
| Method     | Mean mAP | STD mAP  | SDSR-HM  |
+------------+----------+----------+----------+
| FedReID    |  51.8    |  18.4    |  82.2    |
| FedSupWA   |  50.1    |  19.7    |  77.4    |
+------------+----------+----------+----------+
| Difference |   1.7    |   1.3    |   4.8    |
+------------+----------+----------+----------+

The domain-level achievement ratios explain this difference:

+------------+----------+----------+----------+----------+
| Method     | Market   | MSMT17   | CUHK03   | CUHK02   |
+------------+----------+----------+----------+----------+
| FedReID    |  82.2    |  81.5    |  70.2    | 100.0    |
| FedSupWA   |  78.4    |  64.9    |  74.1    | 100.0    |
+------------+----------+----------+----------+----------+

In particular, FedSupWA exhibits substantially lower relative attainment on MSMT17 (64.9% versus 81.5%), despite achieving slightly higher raw performance on CUHK03. The harmonic aggregation makes this weak-domain under-attainment more visible in the overall fairness score.

Second, comparison between SCAFFOLD and FedHRA.

SCAFFOLD achieves a lower mAP standard deviation than FedHRA (14.6% versus 18.7%), which could suggest more uniform raw domain performance when dispersion is considered in isolation. However, its average mAP is substantially lower (32.6% versus 60.1%), and its SDSR-HM is only 49.1 compared with 95.0 for FedHRA.

Table 2. Comparison between SCAFFOLD and FedHRA.

+------------+----------+----------+----------+
| Method     | Mean mAP | STD mAP  | SDSR-HM  |
+------------+----------+----------+----------+
| SCAFFOLD   |  32.6    |  14.6    |  49.1    |
| FedHRA     |  60.1    |  18.7    |  95.0    |
+------------+----------+----------+----------+

Their domain-level achievement ratios further illustrate the difference:

+------------+----------+----------+----------+----------+
| Method     | Market   | MSMT17   | CUHK03   | CUHK02   |
+------------+----------+----------+----------+----------+
| SCAFFOLD   |  43.0    |  63.6    |  32.8    |  83.5    |
| FedHRA     |  93.2    |  91.5    |  95.6    | 100.0    |
+------------+----------+----------+----------+----------+

These results demonstrate that a lower raw standard deviation does not necessarily indicate better fairness, as it may result from uniformly weak performance rather than satisfactory reference-relative achievement. In contrast, SDSR-HM accounts for domain-specific reference performance and penalizes insufficient achievement in weaker domains.

Overall, these two comparisons demonstrate that conventional mean and standard deviation statistics alone may lead to incomplete or misleading fairness interpretations, whereas SDSR-HM provides additional information about how consistently a federated model approaches the achievable performance of each domain.

AuthorsOpenReview ↗

Response to Comment 3: Baseline Training Budgets, Model Selection, and Adaptive Configurations

We thank the reviewer for raising this important concern regarding the fairness and reproducibility of the baseline comparisons. We agree that a fair comparison requires not only consistent training protocols but also appropriate treatment of the method-specific mechanisms of different baselines. We clarify these aspects as follows.

(1) Consistent Training Budgets.

All methods are evaluated under the same four-client federated ReID setting, where Market-1501, MSMT17, CUHK03, and CUHK02 are treated as four independent clients, and all clients participate in every communication round.

For each backbone, all methods use identical data partitions, backbone architectures, local training strategies, ReID loss functions, and evaluation protocols. Specifically, every method is trained for the same number of communication rounds (T), with the same number of local epochs (E) per client in each round. The batch size, input resolution, optimizer, initial learning rate, momentum, weight decay, and learning-rate schedule are also kept consistent across methods using the same backbone.

ResNet50 and ViT are configured separately to accommodate their different computational requirements; however, all methods within each backbone setting follow the same configuration. Since the four datasets have different sizes, the actual number of local optimization steps may differ across clients, but remains identical across methods on the same client. Therefore, no baseline receives additional local training epochs or communication rounds compared with FedHRA.

(2) Consistent Model-Selection and Evaluation Rules.

To avoid introducing advantages through different checkpoint-selection strategies, we apply the same model-selection rule to all compared methods. Specifically, we do not select individual communication rounds according to their test-set performance. Instead, the final results are obtained by averaging the evaluation metrics over the last five communication rounds. This ensures that all methods are evaluated using the same temporal aggregation strategy and prevents test-set-based checkpoint selection from favoring any particular method.

(3) Preservation of Baseline-Specific Adaptive Mechanisms.

We do not disable or replace the original algorithmic components of the compared baselines to enforce identical optimization mechanisms. Instead, each baseline retains its defining objective, update strategy, and adaptive aggregation mechanism.

Specifically, SCAFFOLD maintains its client and server control variates; MOON retains its model-level contrastive regularization; FedProx preserves its proximal regularization term; and q-FFL retains its fairness-aware optimization objective. Similarly, the original fairness-aware aggregation or weighting mechanisms of FedFV, FedHEAL, and FedPW are preserved. The method-specific components of FedPav, FedReID, MEDA, PEFL, SSCU, FedSupWA, and FedARKS are also retained rather than replaced with a common aggregation strategy.

Their method-specific hyperparameters are configured independently according to their respective algorithmic requirements, rather than being forced to share identical parameter values. In contrast, the underlying ReID training protocol and computational budget remain consistent across all methods.

FedHRA likewise retains its three defining components: hard-reliable client scoring, consensus-guided update projection, and reliability-guided fair aggregation, controlled by their corresponding method-specific parameters.

(4) Hyperparameter Selection and Comparison Fairness.

We distinguish between shared training hyperparameters and method-specific hyperparameters. Shared training hyperparameters are fixed across methods within each backbone setting, whereas method-specific parameters are configured according to the corresponding baseline mechanisms.

Hyperparameter selection is conducted using validation data rather than test-set performance, and the same validation-based selection principle is applied to FedHRA and all baselines. This prevents our method from benefiting from additional test-set information or a more favorable model-selection strategy.

Overall, our comparison follows a consistent principle: identical underlying ReID training conditions and evaluation rules, while preserving the original algorithmic characteristics of each baseline. This ensures that the performance differences reported in Table 1 primarily reflect the effectiveness of the respective federated optimization and fairness mechanisms, rather than differences in training budgets or model-selection strategies.

Thank you for the reviewer's response. I believe the reviewer has partially addressed my concerns. However, I still think the paper's contribution standard has not yet reached the bar for ICLR acceptance, so I maintain my score.

Reviewer_4R6JOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper revisits fairness evaluation in federated person re-identification (ReID), arguing that conventional metrics based on cross-domain performance dispersion can be misleading. In particular, low performance variance may reflect uniformly poor retrieval performance rather than equitable achievement, while differences in intrinsic domain difficulty may be confounded with disparities arising from federated optimization. To address these issues, the authors propose Single-Domain Supervised Reference (SDSR), which normalizes each federated model's performance by a protocol-matched single-domain reference and caps the resulting achievement ratio at one. They further introduce SDSR-HM, which aggregates domain-level achievement ratios using the harmonic mean to emphasize underperforming domains. The proposed metrics are evaluated on four ReID datasets—Market-1501, MSMT17, CUHK03, and CUHK02—with ResNet50 and ViT backbones. The experiments compare several general federated learning, federated ReID, and fairness-oriented methods. Additional analyses examine controlled performance degradation and constructed domain heterogeneity. The results suggest that SDSR-HM can distinguish methods that receive similar or misleading rankings under conventional dispersion-based metrics. However, the contribution is primarily an evaluation metric constructed from reference normalization and harmonic aggregation. Further justification of the reference-based fairness objective, stronger validation of reference stability, and improved presentation are needed to establish the general applicability and significance of the proposed approach.

Critical strengths
  1. A relevant problem in fairness evaluation. The paper identifies an important limitation of dispersion-based metrics: similar domain-level performance does not necessarily imply satisfactory or equitable achievement. This distinction is meaningful for federated ReID, where domains can differ substantially in their retrieval characteristics. The proposed reference-relative perspective is a useful direction for complementing conventional metrics.
  2. An interpretable metric with initial empirical evaluation. SDSR-HM has a relatively simple mathematical formulation and can be applied to existing federated ReID methods without changing their training procedures. The evaluation covers four benchmark datasets and two backbone architectures, and the complementary analyses illustrate situations in which raw dispersion and reference-relative attainment lead to different conclusions. These results provide an initial basis for further investigation.
Critical weaknesses
  1. The conceptual justification and novelty of SDSR-HM are insufficiently established. The proposed method combines a capped ratio of federated performance to single-domain reference performance with harmonic-mean aggregation. Although this construction is intuitive, the paper does not sufficiently establish why this particular reference and aggregation rule should be regarded as a reliable measure of fairness rather than one possible measure of relative performance attainment. A single-domain model is not necessarily an estimate of intrinsic domain difficulty or an upper bound on achievable performance. Its performance also reflects the available training data and optimization procedure. Furthermore, capping achievement ratios at one removes distinctions between domains where the federated model exceeds their respective references. These design choices may be reasonable for the intended objective, but their implications and limitations require stronger justification. Consequently, the proposed metric currently appears more convincing as a complementary measure of reference-relative attainment than as a general solution to fairness evaluation.
  2. The empirical validation does not sufficiently establish robustness or generalizability. The evaluation is limited to four ReID domains, and the reported single-domain reference values are based on seed-42 runs with a fixed training budget. The paper does not adequately characterize the variability of these references or demonstrate that the main conclusions remain stable under alternative reference models and training configurations. Moreover, the constructed heterogeneity experiments demonstrate metric sensitivity but do not establish robustness under independently observed domain shifts. Stronger validation with multiple reference-model seeds, sensitivity analyses, and additional real-world domain settings would be needed to support the broader claims. The presentation issues, particularly the crowded main framework diagram, further weaken the overall clarity of the contribution.
Questions for authors
  1. The proposed SDSR-HM score depends directly on single-domain reference performance. How sensitive are the resulting scores and method rankings to the random seed, checkpoint selection, and training budget used to obtain these references? Have the authors evaluated the stability of SDSR-HM using multiple independently trained reference models?
  2. How do the authors distinguish intrinsic domain difficulty from the effects of training-set size, label distribution, and optimization variability when using single-domain performance as the reference? Could the authors clarify the conditions under which SDSR-HM should be interpreted as a fairness measure rather than as reference-relative performance attainment?
  3. The constructed heterogeneity experiments provide useful diagnostic evidence, but their scope is limited. Evaluation on additional independently held-out domains or under naturally occurring domain shifts would help establish whether the proposed metric generalizes beyond the current benchmark settings.
4 comments
AuthorsOpenReview ↗

Response to Comment 1: Stability of Single-Domain References

We thank the reviewer for raising this important concern regarding the reliability of the single-domain references. To investigate reference sensitivity, we examine three aspects: random seed, checkpoint selection, and training budget. For each domain and backbone, we independently train reference models using different configurations while keeping the federated models fixed, thereby isolating the uncertainty introduced by reference construction.

Table 1. Sensitivity analysis of single-domain references.

+------------------+----------------------+----------------------+----------------------+
| Variation Factor | Experimental Setup  | SDSR-HM Variation    | Ranking Consistency  |
+------------------+----------------------+----------------------+----------------------+
| Random Seed      | 40, 2026, 3888       | Std: 0.4-0.6 pp      | Spearman: 0.95-0.96  |
|                  |                      |                      | Kendall: 0.90-0.92   |
+------------------+----------------------+----------------------+----------------------+
| Checkpoint       | Epoch 60, 70, 80     |                      |                      |
| ResNet50-mAP     |                      | 0.73 pp              |                      |
| ResNet50-Rank-1  |                      | 0.78 pp              |                      |
| ViT-mAP          |                      | 1.08 pp              |                      |
| ViT-Rank-1       |                      | 0.50 pp              |                      |
+------------------+----------------------+----------------------+----------------------+
| Training Budget  | 60, 80, 100 epochs   | Relatively small     |                      |
|                  |                      | variations           |                      |
+------------------+----------------------+----------------------+----------------------+

First, under different random seeds, the standard deviations of SDSR-HM remain within 0.4-0.6 percentage points across the evaluated backbone-metric combinations. The corresponding method rankings also remain highly consistent, with Spearman correlations of 0.95-0.96 and Kendall correlations of 0.90-0.92.

Second, checkpoint sensitivity is evaluated using reference models from epochs 60, 70, and 80. The SDSR-HM variations are 0.73 and 0.78 percentage points for ResNet50 mAP and Rank-1, respectively, and 1.08 and 0.50 percentage points for ViT. These results indicate that reasonable checkpoint variations have a limited influence on the resulting fairness scores.

Third, we evaluate reference models trained for 60, 80, and 100 epochs. The resulting SDSR-HM scores exhibit relatively small variations, suggesting that moderate changes in training budgets do not substantially affect the overall evaluation.

Overall, these results demonstrate that SDSR-HM scores and method rankings remain reasonably stable against variations in reference initialization, checkpoint selection, and training duration. We will incorporate these sensitivity analyses and further clarify the reference construction protocol in the revised manuscript.

AuthorsOpenReview ↗

Response to Comment 2: Interpretation of Domain References and Fairness

We appreciate the reviewer for raising this important question regarding the interpretation of single-domain references and the fairness objective of SDSR-HM.

(1) Distinguishing intrinsic domain difficulty from training-related factors.

We acknowledge that the performance of a single-domain supervised model is jointly influenced by several factors, including intrinsic visual domain characteristics, training-set size, identity-label distribution, and optimization conditions. Therefore, a single-domain reference cannot independently identify intrinsic domain difficulty or serve as a pure estimate of the optimal performance achievable on a domain.

In our framework, we do not claim to causally disentangle these factors. Instead, we define the single-domain reference as a protocol-conditioned empirical reference, which captures the performance achievable by an independently trained model under the corresponding domain-specific data conditions.

To improve comparability, we maintain consistent backbone architectures, ReID training protocols, evaluation metrics, and domain-specific test splits between the reference and federated models. We also investigate reference sensitivity to random seeds, checkpoint selection, and training budgets to reduce the influence of optimization-related randomness.

Importantly, matching the training protocol reduces optimization-related differences but does not eliminate the effects of unequal training-set sizes or identity distributions. Consequently, our references represent domain-specific empirical performance under the available data conditions, rather than isolated measurements of intrinsic domain difficulty.

(2) When should SDSR-HM be interpreted as a fairness metric rather than relative performance attainment?

The distinction depends on the evaluation objective and the comparability of the reference construction.

SDSR-HM is designed to evaluate whether a federated model achieves balanced performance relative to the independently established references of different domains. It does not simply measure the performance improvement of a federated model over another model. Instead, it examines the consistency of reference-relative performance attainment across heterogeneous domains.

We consider SDSR-HM appropriate as a cross-domain fairness measure under four conditions. First, reference and federated models should follow consistent evaluation protocols, including the same backbone architecture, evaluation metrics, and domain-specific test splits. Second, single-domain references should be sufficiently stable with respect to random initialization, checkpoint selection, and training duration. Third, the same fixed domain-specific references should be used when evaluating all federated methods, ensuring that score differences are not caused by changing reference models. Fourth, the evaluation objective should be to assess balanced performance attainment across heterogeneous domains rather than absolute retrieval accuracy or individual client-level performance improvements.

Under these conditions, SDSR-HM provides a measure of cross-domain fairness in terms of balanced reference-relative achievement. However, it should not be interpreted as proving that the intrinsic difficulty of all domains has been equalized or that differences in data availability and identity distributions have been eliminated.

We will explicitly clarify this distinction, the intended interpretation, and the limitations of SDSR-HM in the revised manuscript.

AuthorsOpenReview ↗

Response to Comment 3: Generalization Beyond Constructed Heterogeneity

We thank the reviewer for this valuable suggestion. We agree that the constructed heterogeneity experiments alone cannot fully establish the generalizability of SDSR-HM beyond the original benchmark settings. To address this concern, we extend our evaluation to PRID2011, an independently held-out ReID dataset that is excluded from the original federated training process.

Specifically, PRID2011 contains two camera views (Camera A and Camera B), with 200 identities appearing in both views. Following its standard single-shot protocol, we divide these identities into 100 training identities and 100 testing identities. We evaluate two cross-camera retrieval directions, A-to-B and B-to-A, to examine the effect of naturally occurring camera-dependent distribution differences.

The federated models are trained exclusively on the original four datasets (Market-1501, MSMT17, CUHK03, and CUHK02), without using PRID2011 images or labels for federated training or hyperparameter selection. We then directly evaluate the trained models on PRID2011 without target-domain fine-tuning. For reference construction, we independently train a single-domain supervised model using the PRID2011 training identities under the same backbone and ReID training protocol. Its performance is evaluated separately in both retrieval directions to obtain direction-specific reference scores.

Table 1. Cross-camera generalization and SDSR-HM evaluation on the independently held-out PRID2011 dataset.

+----------+--------+--------+----------+--------+--------+----------+
| Method   | mAP A-B| mAP B-A| SDSR-HM  | R1 A-B | R1 B-A | SDSR-HM  |
|          |        |        | (mAP)    |        |        | (R-1)    |
+----------+--------+--------+----------+--------+--------+----------+
| FedHRA   |  42.5  |  36.8  |  75.4    |  56.9  |  50.3  |  76.4    |
| FedFV    |  39.4  |  33.5  |  69.2    |  52.2  |  46.0  |  70.0    |
| PEFL     |  37.8  |  31.2  |  65.4    |  49.5  |  42.8  |  65.7    |
| FedReID  |  36.5  |  30.8  |  63.9    |  47.8  |  42.0  |  64.0    |
| SCAFFOLD |  28.4  |  23.1  |  48.8    |  38.0  |  31.5  |  49.3    |
+----------+--------+--------+----------+--------+--------+----------+

The results indicate that FedHRA achieves the highest reference-relative performance in both cross-camera directions, with SDSR-HM scores of 75.4 for mAP and 76.4 for Rank-1. The performance differences between A-to-B and B-to-A also demonstrate the asymmetric effects of natural camera-dependent distribution shifts.

Importantly, this experiment differs from the constructed heterogeneity analysis in Figure 3(b). Rather than manually specifying degradation factors, it evaluates federated models on an independently collected dataset with naturally occurring camera and appearance variations. The use of direction-specific reference scores further allows SDSR-HM to account for differences in retrieval difficulty between the two evaluation directions.

We acknowledge that evaluation on one additional dataset does not establish universal generalization across all ReID environments. Nevertheless, this experiment provides complementary evidence that the proposed reference-based evaluation framework can be extended beyond the original four-domain benchmark and applied to naturally occurring cross-camera distribution shifts. We will clarify the experimental scope and corresponding limitations in the revised manuscript.

Thank you for the detailed response. I appreciate the authors' efforts to address the review comments. These revisions clarify some of my concerns and improve the completeness of the paper. However, considering the paper's contributions, experimental quality, and remaining limitations, I decide to maintain my original rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.