- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors propose FedPID to evaluate client contributions in FL. By utilizing predictions on a common validation set and applying PID, it decomposes each client's prediction information into Unique, Redundant, and Synergistic components relative to other clients. These components are analyzed per class and then combined into scores for model aggregation. Through experiments comparing contribution estimates under various conditions of data size, noise level, and class distribution, the study demonstrates improved aggregation performance in certain settings.
Critical strengths
1.. The method provides class-level diagnostic profiles that help explain the informational roles of individual clients. 2. PID-based aggregation improves performance under noisy-client settings, including over validation-based weighting methods.
Critical weaknesses
-
Weak motivation. The paper argues that client contributions should be evaluated through three separate components, but the motivation for this separation is weak. The authors should explain which aspects of contribution matter in which scenarios and provide experimental evidence. Currently, the paper mainly claims that this distinction is necessary without sufficiently supporting why it is beneficial.
-
Overstated limitations of existing methods. A method does not necessarily fail to capture the aspects discussed in the paper simply because it produces a single scalar score. Although one scalar metric may not evaluate all three components separately, different metrics could evaluate these components. Three separate metrics could therefore be used. The paper does not sufficiently establish why all three components need to be included within a single contribution metric.
-
Inability to measure actual aggregation contributions. PID-based client contribution evaluation does not measure how much a client contributes to performance improvement through aggregation. It does not establish which clients actually contribute to improving the final aggregated model’s performance.
-
Insufficient validation of contribution measurement accuracy. The paper does not compare contribution evaluation methods against an objective reference criterion. For example, it assumes that clients with more data should receive higher contribution scores and then validates PID by showing that its scores follow this assumption. Agreement with this expected pattern alone is insufficient to establish that PID measures client contributions more accurately than existing methods.
Questions for authors
See weaknesses
8 comments
(2/4) Point 2. Why unique, redundant and synergistic information make client contribution interpretable in a way that three scalar competitors could not provide.
State-of-the-art scalar methods fail to diagnose a specialist's information contributions in non-IID settings. SV and LOO both assess changes in a chosen performance utility, differing in which coalitions they consider; GTG approximates Shapley values, and GCS measures gradient update alignment. As we have shown in our experiments, there are scenarios where none of them is able to correctly identify client contributions. For instance, in the non-IID setting reported in Table 1, all four methods assign to C1 a near-zero or the lowest value. Neither their individual scores nor a combination of three of them reveals that C1 supplies unique information about classes 0–1 that its peers lack. FedPID's class-wise components identify precisely this role (Figures 1, 3 and 7).
PID derives these components from a complete information-theoretic framework. FedPID's three components are not separately designed scores, and the fact that they are three is not a coincidence. For each class, the two-source PID decomposition is . The term is information unique to the peers; the remaining three terms account for all the client-related components in this two-source formulation. Unique, redundant and synergistic information are therefore parts of one decomposition of the same predictive information, with defined meanings for the same client, peers and class target. Reporting SV, LOO and GCS together combines measures of performance or alignment; it does not provide this decomposition. FedPID thereby tackles the problem of contribution evaluation from a novel, information theory-centric lens, which allows us to explain for the first time whether a client supplies information missing from peers, shares information with them, or contributes information available only jointly.
(3/4) Point 3. Informational contribution and aggregation performance.
FedPID measures client contribution from an information-theoretic perspective: how much predictive information a client provides about each class, and whether it is unique, shared or available only jointly with peers. This is indeed different from some of the state-of-the-art competitors which assign value to clients based on aggregation performance improvement: for instance, Shapley values (based on validation accuracy) average marginal accuracy gains across coalitions, and leave-one-out (LOO) measures the accuracy change after removing a client under a fixed training procedure. Crucially, one contribution of our paper is precisely to demonstrate that a low or negative performance-based score does not establish that a client lacks information useful for aggregation.
We directly test this distinction in the non-IID setting with an additional experiment, where we compare FedAvg with all five clients (A), FedAvg excluding C1 throughout training (B), Shapley-based aggregation using the SV scores from Table 1 (C), and FedPID$_R$ with all five clients using (D). For C, we normalise the scores by subtracting the minimum SV score (the negative value of for C1) from all scores and normalising the resulting values so that they sum to one (C1 thus receives zero aggregation weight, similarly to B).
The results averaged over three seeds are reported below. In A, FedAvg assigns equal aggregation weights to all clients, so C1's specialist signal is diluted and effectively lost during parameter update averaging. Removing C1 improves FedAvg's overall accuracy, but A, B and C all have zero accuracy on classes 0–1, which only C1 holds. Instead, our approach D correctly estimates the contribution provided by C1: the final model is capable of detecting both classes and outperforms A by and C by percentage points. Thus, using validation accuracy improvement to define contribution does not guarantee a weighting rule that exploits all useful information. Instead, FedPID's profiles identify C1's distinctive class information, and their use for weighting improves the final model beyond the tested Shapley-based policy. (Note: the results below are test accuracies (%, mean sample standard deviation over three seeds).)
| Accuracy | A: FedAvg C1–C5 | B: Remove C1 | C: Shapley | D: FedPID$_R$ |
|---|---|---|---|---|
| Overall | ||||
| 0: airplane | ||||
| 1: automobile | ||||
| 2: bird | ||||
| 3: cat | ||||
| 4: deer | ||||
| 5: dog | ||||
| 6: frog | ||||
| 7: horse | ||||
| 8: ship | ||||
| 9: truck |
(1/4) Point 1. Motivation and benefit of estimating the three separate components.
FedPID's motivation is to identify how a client provides predictive information: whether it supplies information unavailable from its peers, information shared with them, or information available only when its predictions and those of its peers are considered jointly. Identifying these contributions is crucial in several federated scenarios, as described below:
Unique information matters when only one client has samples of particular classes. In the CIFAR-10 non-IID setting, all five clients have 5,000 examples, but only C1 holds samples of classes 0–1, bringing to the federation absolutely crucial information. However, as shown in Table 1, state-of-the-art contribution evaluation approaches like SV, LOO and GTG assign to C1 near-zero or negative contribution scores, and GCS the lowest among the clients. Our proposed scalar FedPID$_F$ assigns to C1 the highest score, although it is not substantially higher than that of C2 (0.233 vs 0.225). These scalar scores do not reveal that C1 supplies information about classes 0–1 that its peers lack. Instead, by looking at individual FedPID components, we are able to locate C1's unique information in those classes (as shown in Figures 1, 3 and 7). The practical benefit is to identify which client's information needs to be retained and used to identify samples of these classes, even when the scalar baselines rank that client last. Additionally, point 3 below shows that using FedPID's profiles for aggregation improves overall accuracy.
Redundancy distinguishes overlapping expertise from additional information. In the CIFAR-10 IID setting, SV and GCS assign nearly uniform scores to all clients (Table 1). These scalar outputs do not explain whether clients provide different predictive information or largely repeat one another's signal. FedPID's profiles show that redundancy dominates while unique information remains zero (Figures 1–2): the clients are informative, but much of their information is shared. As shown in Figure 2, our method also estimates synergy decreasing as redundancy increases during training. Separating these components distinguishes information available only through joint predictions from information already shared between clients, revealing whether their contributions involve complementarity or overlap. This helps to assess whether the participating group brings different types of information (e.g. samples of different classes) or mainly repeats existing information. This allows us, for instance, to better interpret small marginal gains.
Specialist information can be unique or shared depending on the participating peers. In Table 11(b), the specialist's unique information assigned by FedPID in the 20-client pool falls from 0.1285±0.0105 to 0.0080±0.0022 bits when another specialist participates, while redundancy rises from zero to 0.1059±0.0037 bits; the 50-client pool shows the same pattern. We therefore learn that a specialist's information is mainly unique when no peer specialises in the same classes, but mainly shared when another such specialist participates. For participation or weighting decisions, this identifies whether the active group obtains distinctive information from that specialist or already has another source of similar information. Separating unique and redundant information makes this change in role explicit. A single contribution score does not say whether the specialist's information is unique or shared.
In summary, FedPID's profiles provide insights on client contributions that scalar competitors simply cannot provide, and do so within the formal framework of information theory. These insights are crucial for decision-making processes involved in training federated learning models.