acceptodds
Under review as a conference paper at ICLR 2027

Interpretable Client Contribution Evaluation in Federated Learning via Partial Information Decomposition

Abstract

In federated learning (FL), multiple clients collaboratively train a shared model without sharing their raw data. Understanding how much each client contributes to the global model is central to fair incentive design, principled client selection, and the detection of unreliable or harmful participants. Yet client contribution is typically summarised by a single scalar utility, which conflates fundamentally different sources of value: information that is unique to a client, information that is duplicated elsewhere in the federation, and information that emerges only when several clients are combined. We propose FedPID, an information-theoretic framework for client contribution evaluation based on Partial Information Decomposition (PID), characterising each client's value in terms of these three components rather than a single number. FedPID builds class-conditioned PIDs for each client from predictions on a shared server-side validation set, producing round-wise profiles of unique, redundant, and synergistic information that can be inspected directly as a diagnostic, used to compare clients, or converted into contribution-guided aggregation weights. Across heterogeneous client settings on CIFAR-10, Fed-ISIC, and ImageNet-100, FedPID recovers expected contribution orderings from first-round profiles, identifies specialist clients that other scalar methods undervalue, and separates useful clients from noisy ones. As one downstream use, PID-derived aggregation weights improve performance in noisy-client settings compared with baselines. FedPID thus reframes client contribution as an informational composition rather than a single opaque score, providing both an interpretable diagnostic and a control signal for aggregation.

Primary area:
interpretability and explainable AI
Keywords:
Federated Learning, Partial Information Decomposition, Client Contribution Evaluation, Information Theory, Interpretable Machine Learning

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 2 reviews, above 68% of the others.
Reviewer_ytSEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose FedPID to evaluate client contributions in FL. By utilizing predictions on a common validation set and applying PID, it decomposes each client's prediction information into Unique, Redundant, and Synergistic components relative to other clients. These components are analyzed per class and then combined into scores for model aggregation. Through experiments comparing contribution estimates under various conditions of data size, noise level, and class distribution, the study demonstrates improved aggregation performance in certain settings.

Critical strengths

1.. The method provides class-level diagnostic profiles that help explain the informational roles of individual clients. 2. PID-based aggregation improves performance under noisy-client settings, including over validation-based weighting methods.

Critical weaknesses
  1. Weak motivation. The paper argues that client contributions should be evaluated through three separate components, but the motivation for this separation is weak. The authors should explain which aspects of contribution matter in which scenarios and provide experimental evidence. Currently, the paper mainly claims that this distinction is necessary without sufficiently supporting why it is beneficial.

  2. Overstated limitations of existing methods. A method does not necessarily fail to capture the aspects discussed in the paper simply because it produces a single scalar score. Although one scalar metric may not evaluate all three components separately, different metrics could evaluate these components. Three separate metrics could therefore be used. The paper does not sufficiently establish why all three components need to be included within a single contribution metric.

  3. Inability to measure actual aggregation contributions. PID-based client contribution evaluation does not measure how much a client contributes to performance improvement through aggregation. It does not establish which clients actually contribute to improving the final aggregated model’s performance.

  4. Insufficient validation of contribution measurement accuracy. The paper does not compare contribution evaluation methods against an objective reference criterion. For example, it assumes that clients with more data should receive higher contribution scores and then validates PID by showing that its scores follow this assumption. Agreement with this expected pattern alone is insufficient to establish that PID measures client contributions more accurately than existing methods.

Questions for authors

See weaknesses

8 comments

(1/4) Point 1. Motivation and benefit of estimating the three separate components.

FedPID's motivation is to identify how a client provides predictive information: whether it supplies information unavailable from its peers, information shared with them, or information available only when its predictions and those of its peers are considered jointly. Identifying these contributions is crucial in several federated scenarios, as described below:

Unique information matters when only one client has samples of particular classes. In the CIFAR-10 non-IID setting, all five clients have 5,000 examples, but only C1 holds samples of classes 0–1, bringing to the federation absolutely crucial information. However, as shown in Table 1, state-of-the-art contribution evaluation approaches like SV, LOO and GTG assign to C1 near-zero or negative contribution scores, and GCS the lowest among the clients. Our proposed scalar FedPID$_F$ assigns to C1 the highest score, although it is not substantially higher than that of C2 (0.2330.233 vs 0.2250.225). These scalar scores do not reveal that C1 supplies information about classes 0–1 that its peers lack. Instead, by looking at individual FedPID components, we are able to locate C1's unique information in those classes (as shown in Figures 1, 3 and 7). The practical benefit is to identify which client's information needs to be retained and used to identify samples of these classes, even when the scalar baselines rank that client last. Additionally, point 3 below shows that using FedPID's profiles for aggregation improves overall accuracy.

Redundancy distinguishes overlapping expertise from additional information. In the CIFAR-10 IID setting, SV and GCS assign nearly uniform scores to all clients (Table 1). These scalar outputs do not explain whether clients provide different predictive information or largely repeat one another's signal. FedPID's profiles show that redundancy dominates while unique information remains zero (Figures 1–2): the clients are informative, but much of their information is shared. As shown in Figure 2, our method also estimates synergy decreasing as redundancy increases during training. Separating these components distinguishes information available only through joint predictions from information already shared between clients, revealing whether their contributions involve complementarity or overlap. This helps to assess whether the participating group brings different types of information (e.g. samples of different classes) or mainly repeats existing information. This allows us, for instance, to better interpret small marginal gains.

Specialist information can be unique or shared depending on the participating peers. In Table 11(b), the specialist's unique information assigned by FedPID in the 20-client pool falls from 0.1285±0.01050.1285\pm0.0105 to 0.0080±0.00220.0080\pm0.0022 bits when another specialist participates, while redundancy rises from zero to 0.1059±0.00370.1059\pm0.0037 bits; the 50-client pool shows the same pattern. We therefore learn that a specialist's information is mainly unique when no peer specialises in the same classes, but mainly shared when another such specialist participates. For participation or weighting decisions, this identifies whether the active group obtains distinctive information from that specialist or already has another source of similar information. Separating unique and redundant information makes this change in role explicit. A single contribution score does not say whether the specialist's information is unique or shared.

In summary, FedPID's profiles provide insights on client contributions that scalar competitors simply cannot provide, and do so within the formal framework of information theory. These insights are crucial for decision-making processes involved in training federated learning models.

(2/4) Point 2. Why unique, redundant and synergistic information make client contribution interpretable in a way that three scalar competitors could not provide.

State-of-the-art scalar methods fail to diagnose a specialist's information contributions in non-IID settings. SV and LOO both assess changes in a chosen performance utility, differing in which coalitions they consider; GTG approximates Shapley values, and GCS measures gradient update alignment. As we have shown in our experiments, there are scenarios where none of them is able to correctly identify client contributions. For instance, in the non-IID setting reported in Table 1, all four methods assign to C1 a near-zero or the lowest value. Neither their individual scores nor a combination of three of them reveals that C1 supplies unique information about classes 0–1 that its peers lack. FedPID's class-wise components identify precisely this role (Figures 1, 3 and 7).

PID derives these components from a complete information-theoretic framework. FedPID's three components are not separately designed scores, and the fact that they are three is not a coincidence. For each class, the two-source PID decomposition is I(Y(c);Zk(c),Z−k(c))=Uk+U−k+R+SI(Y^{(c)};Z_k^{(c)},Z_{-k}^{(c)})=U_k+U_{-k}+R+S. The term U−kU_{-k} is information unique to the peers; the remaining three terms account for all the client-related components in this two-source formulation. Unique, redundant and synergistic information are therefore parts of one decomposition of the same predictive information, with defined meanings for the same client, peers and class target. Reporting SV, LOO and GCS together combines measures of performance or alignment; it does not provide this decomposition. FedPID thereby tackles the problem of contribution evaluation from a novel, information theory-centric lens, which allows us to explain for the first time whether a client supplies information missing from peers, shares information with them, or contributes information available only jointly.

(3/4) Point 3. Informational contribution and aggregation performance.

FedPID measures client contribution from an information-theoretic perspective: how much predictive information a client provides about each class, and whether it is unique, shared or available only jointly with peers. This is indeed different from some of the state-of-the-art competitors which assign value to clients based on aggregation performance improvement: for instance, Shapley values (based on validation accuracy) average marginal accuracy gains across coalitions, and leave-one-out (LOO) measures the accuracy change after removing a client under a fixed training procedure. Crucially, one contribution of our paper is precisely to demonstrate that a low or negative performance-based score does not establish that a client lacks information useful for aggregation.

We directly test this distinction in the non-IID setting with an additional experiment, where we compare FedAvg with all five clients (A), FedAvg excluding C1 throughout training (B), Shapley-based aggregation using the SV scores from Table 1 (C), and FedPID$_R$ with all five clients using U+0.05R+0.05SU+0.05R+0.05S (D). For C, we normalise the scores by subtracting the minimum SV score (the negative value of −0.005-0.005 for C1) from all scores and normalising the resulting values so that they sum to one (C1 thus receives zero aggregation weight, similarly to B).

The results averaged over three seeds are reported below. In A, FedAvg assigns equal aggregation weights to all clients, so C1's specialist signal is diluted and effectively lost during parameter update averaging. Removing C1 improves FedAvg's overall accuracy, but A, B and C all have zero accuracy on classes 0–1, which only C1 holds. Instead, our approach D correctly estimates the contribution provided by C1: the final model is capable of detecting both classes and outperforms A by 7.027.02 and C by 5.285.28 percentage points. Thus, using validation accuracy improvement to define contribution does not guarantee a weighting rule that exploits all useful information. Instead, FedPID's profiles identify C1's distinctive class information, and their use for weighting improves the final model beyond the tested Shapley-based policy. (Note: the results below are test accuracies (%, mean ±\pm sample standard deviation over three seeds).)

AccuracyA: FedAvg C1–C5B: Remove C1C: ShapleyD: FedPID$_R$
Overall69.83±0.4569.83\pm0.4571.60±0.4271.60\pm0.4271.58±0.6071.58\pm0.6076.86±1.0276.86\pm1.02
0: airplane0.00±0.000.00\pm0.000.00±0.000.00\pm0.000.00±0.000.00\pm0.0062.71±7.7662.71\pm7.76
1: automobile0.00±0.000.00\pm0.000.00±0.000.00\pm0.000.00±0.000.00\pm0.0053.90±12.9053.90\pm12.90
2: bird81.14±6.9281.14\pm6.9287.43±0.1487.43\pm0.1486.76±0.4486.76\pm0.4475.05±2.2775.05\pm2.27
3: cat86.29±2.8986.29\pm2.8977.43±2.2577.43\pm2.2577.90±3.7377.90\pm3.7372.76±8.5672.76\pm8.56
4: deer90.86±1.0890.86\pm1.0889.81±0.8289.81\pm0.8291.14±0.4991.14\pm0.4986.29±1.3686.29\pm1.36
5: dog74.14±0.5274.14\pm0.5282.00±0.8982.00\pm0.8982.71±0.4982.71\pm0.4973.52±4.9473.52\pm4.94
6: frog87.10±2.8987.10\pm2.8992.43±0.5292.43\pm0.5291.90±1.1591.90\pm1.1587.48±2.3287.48\pm2.32
7: horse87.52±1.2087.52\pm1.2092.62±0.8692.62\pm0.8691.24±1.3091.24\pm1.3080.24±3.6780.24\pm3.67
8: ship94.57±0.8794.57\pm0.8796.38±0.7996.38\pm0.7996.86±0.4396.86\pm0.4387.57±3.7187.57\pm3.71
9: truck96.71±0.3896.71\pm0.3897.86±0.3897.86\pm0.3897.29±0.4397.29\pm0.4389.05±1.9489.05\pm1.94
Reviewer_9ZcjOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes FedPID, an interpretable framework for evaluating client contributions in federated learning using Partial Information Decomposition (PID). Unlike conventional scalar-based methods, FedPID decomposes client contributions into unique, redundant, and synergistic information through class-wise analysis of validation predictions. The resulting contribution profiles support fine-grained interpretation and contribution-aware model aggregation. Experiments on CIFAR-10, Fed-ISIC, and ImageNet-100 demonstrate its effectiveness in identifying specialist and noisy clients, with improved aggregation performance in selected noisy settings. The key novelty lies in reformulating client valuation as an interpretable information decomposition rather than a single utility score.

Critical strengths
  1. FedPID exposes class-specific unique information and separates it from redundancy and synergy, giving a more informative view of specialists than leave-one-out or a single held-out loss. The non-IID CIFAR-10 examples make the intended interpretation concrete.
  2. The paper studies several heterogeneity mechanisms and includes Fed-ISIC and ImageNet-100, free-rider/data-inflation tests, partial participation, validation-set stress tests, and runtime measurements.
Critical weaknesses
  1. The “contribution” is a relative, estimator-dependent quantity. Because Z−kZ_{-k} is the current ordered peer tuple, the unique/redundant/synergistic values change with the participating set, participation rate, Imin⁡I_{\min} choice, and probability bins. The reported K=20/50K=20/50 scale tests still use only m=10m=10 participants per round, so they do not establish the finite-sample behavior of the high-dimensional tuple. The paper does not give effective joint-state counts, confidence intervals, or a systematic comparison with alternative redundancy measures. Summing one-vs-rest quantities with class priors also needs a careful qualification before it is described as a multi-class contribution.
  2. The aggregation claims are not tested against strategic clients, and the supplied artifact is incomplete.
Questions for authors
  1. Please state the exact units, normalization denominators, and aggregation formula for each profile component. What conditions justify calling the result a contribution rather than a relative diagnostic?
  2. What happens when a client deliberately optimizes validation-set unique information, submits random/backdoor updates, or fabricates confidence values? Please test at least one strategic or Byzantine client and specify a pre-aggregation safeguard if the profile can be manipulated.
  3. For genuinely large partial participation (K≫50K\gg 50 and varying mm), what are the sample and computational scaling laws of the tuple estimator? How are unequal participation frequencies, cold-start clients, and newly joining clients corrected in the cumulative score?
9 comments

(1/4)

We thank the reviewer for their careful assessment and helpful comments on the interpretation, estimation reliability and practical use of FedPID. We have done our best to address all the weaknesses and questions raised, as detailed below.

Point 1. What contribution means, its units and its dependence on peers (W1/Q1).

FedPID defines contribution through the predictive information supplied by a client relative to its participating peers. FedPID's components identify information unique to that client, shared with peers, or available only through their joint predictions. This is the intended informational meaning of contribution. Peer dependence is also present in established contribution measures: Shapley values are defined for a player set, and round-wise GTG-Shapley and leave-one-out evaluations change based on the participating clients. This dependence is necessary to distinguish information the federation already has from information a client adds. Table 11(b) illustrates this directly: in the K=20K=20 specialist setting, unique information falls from 0.1285±0.01050.1285\pm0.0105 to 0.0080±0.00220.0080\pm0.0022 bits when another specialist participates, while redundancy rises from zero to 0.1059±0.00370.1059\pm0.0037 bits. FedPID explains this change in the client's contribution.

Raw information, displayed fractions and scalar scores have separate definitions. All raw components use base-2 logarithms and are measured in bits. Equation (6) forms the whole-client profile by weighting each class-level component by the validation class prior πc=Nc/N\pi_c=N_c/N:

R_k=\sum_c\pi_c R_{k,c},\qquad S_k=\sum_c\pi_c S_{k,c},$$ where $N_c$ is the number of validation examples belonging to class $c$ and $N$ is the total size of the validation set. In Figures 1–3 and 7, each displayed PID component is normalised by the corresponding total information: the sum of unique information from the client, unique information from its peers, redundancy and synergy. Each displayed value is therefore a fraction of this total. We will make this normalisation explicit in the figure captions. For the reported client contribution evaluation comparisons, we combine the components into a single score, $s_k=U_k+0.15R_k+0.15S_k$, and divide by the total score of the participating clients, $q_k=s_k/\sum_{\ell\in\mathcal S}s_\ell$. Thus $q_k$ expresses client $k$'s share of the total score in that round, and these shares sum to one. Point 5 below clarifies the distinction between the participation average and the accumulated score used for aggregation. **FedPID correctly identifies useful client contribution and improves aggregation.** In the CIFAR-10 feature-and-label-noise setting, FedPID identifies the clean client and assigns progressively lower scores as corruption increases (Table 1). FedPID\$\_F\$ also achieves $84.75\\%$ aggregation accuracy, compared with $76.47\\%$ for the strongest baseline (Table 3). More crucially, in the non-IID specialist setting, **FedPID alone among the compared methods** correctly identifies C1's unique contribution to classes 0–1 (Figures 3 and 7), whereas SV assigns C1 a near-zero negative score (Table 1). As detailed in **Point 3 of our response to Reviewer ytSE**, FedPID\$\_R\$ achieves $76.86\pm1.02\\%$ overall accuracy, outperforming the tested SV-based aggregation rule ($71.58\pm0.60\\%$) by $5.28$ percentage points. It also manages to detect both specialist classes, for which SV-based aggregation achieves zero accuracy.

(2/4)

Point 2. PID definition, probability bins and multi-class contribution (W1).

Our aim is to establish the usefulness of PID as a framework for interpreting client contribution, rather than exhaustively benchmark all PID definitions. Other PID measures can also be used within this framework to calculate the unique, redundant and synergistic components. The resulting component values can differ between measures. Appendix A.3 already compares Imin⁡I_{\min}, BROJA and SxPID, showing the quantity-skew setting as an example in Figure 5. We use Imin⁡I_{\min}, the original Williams–Beer definition, for its non-negative components and direct computation from specific information. Figure 5 shows that BROJA has a very similar trend to Imin⁡I_{\min}, but it requires optimisation over distributions and it is more computationally expensive than the direct Imin⁡I_{\min} calculation. SxPID can produce negative components, which would require additional choices to use separately weighted U/R/SU/R/S terms as non-negative aggregation weights. These considerations explain our implementation choice without asserting that one definition is universally preferable.

Probability binning defines the representation used for PID. This choice is part of the contribution definition, so FedPID values depend on the bin count. Section 6 already acknowledges that “Discretisation trades confidence detail for denser empirical counts”. We assess this dependence in Appendix D.5: Table 16 compares 2, 4 and 8 bins. While score magnitudes change (for instance in the feature-and-label-noise setting), the qualitative pattern of higher contributions for cleaner clients is retained.

FedPID combines class-level PID components into a multi-class contribution profile. For each class, it computes a PID for distinguishing that class from the remaining classes. Equation (6) in Section 4.2 then combines these class-level components using validation class priors into a whole-client profile covering all classes. The text following Table 18 already clarifies that FedPID's components are weighted sums of one-vs-rest information terms. Thus “multi-class contribution” refers to this class-weighted evaluation across all classes. We will make the connection to this existing explanation explicit in Section 4.2.

(3/4)

Point 3. Reliability with more participating clients and validation data (W1/Q3).

We vary the number of participating clients and validation examples to examine the tuple estimator. Here KK is the total client pool, mm is the number of clients participating in a training round, and NN is the number of validation samples. Table 11(a) already includes both m=20m=20 and m=50m=50 for K=20K=20 and K=50K=50 pools, respectively (full-participation cases), alongside m=10m=10 (partial participation). Below we report an additional experiment: using CIFAR-10 IID predictions from the K=50K=50 pool, we select subsets of clients' saved predictions, varying m∈5,10,20,50m\in\\{5,10,20,50\\} and N∈250,500,1000,3000N\in\\{250,500,1000,3000\\} at rounds 1 and 40 across three training seeds.

For each evaluated client and target class, a joint peer state is the vector of its peers' binned predictions for that class. We count the observed joint peer states across the NN validation images and the percentage of images whose state occurs only once (singleton images). These quantities are averaged over clients, classes and three seeds. The CPU time column averages rounds 1 and 40 and three seeds; it measures PID calculation from saved predictions and excludes model inference.

mmNNObserved joint peer states (Round 1)Observed joint peer states (Round 40)Singleton images (%) (Round 1)Singleton images (%) (Round 40)Mean PID time (s)
525027.128.33.776.910.012
53,00050.183.70.290.910.126
10250101.449.629.5517.250.029
103,000510.6399.89.7610.510.351
20250166.562.461.1123.260.077
203,0001533.5617.544.8619.271.023
50250204.773.678.9428.220.308
503,0002209.3776.270.7424.864.271

Larger validation sets reduce the percentage of singleton images. For every tested mm, the percentage of singleton images decreases as NN increases, both after 1 and 40 rounds. For example, with m=50m=50, increasing NN from 250 to 3000 reduces it from 78.9478.94\\% to 70.7470.74\\% in round 1 and from 28.2228.22\\% to 24.8624.86\\% in round 40. These results quantify the finite-sample behaviour acknowledged in Section 6; they do not indicate a contradiction in the PID formulation. Raw components also change with validation size: at m=5m=5 in round 1, mean synergy changes from 0.046020.04602 to 0.018900.01890 bits as NN increases from 250 to 3000. For complementary results showing how the raw information components change with validation-set size, please see the table in Point 2 of our response to Reviewer LqdP.

Bootstrap resampling supports a stable PID profile in this IID run. For the five-client IID models at round 40 (seed 42), we recompute PID on 2,000 resamples of the 3,000 validation images. Each resample draws 300 images per class with replacement, keeping each selected image's label and all clients' predictions together; the models remain fixed. For C1, redundancy is 0.38320.3832 bits with an approximate 95% percentile confidence interval (CI) of [0.3757,0.3934][0.3757,0.3934], synergy is 0.004490.00449 with a CI of [0.00237,0.00617][0.00237,0.00617], and unique information is 0.0000.000. These intervals indicate a highly stable estimate of the overall profile under validation resampling in this setting/run. We will include the full m×Nm\times N results, raw components and all clients' intervals in Appendix D.3.

The number of participating clients determines the per-round computational cost. Even when K≫50K\gg50, only the mm participating clients enter the round's PID calculation, so each client's peer tuple has m−1m-1 coordinates. Increasing the total pool KK while holding mm fixed therefore does not increase the tuple dimension or the per-round computational complexity. As stated in Appendix D.3, each PID measurement requires mm validation passes and mCmC two-source PID calculations, where CC is the number of classes, with O(Cm2Nlog⁡N)O(Cm^2N\log N) work for grouping the observed predictions by row sorting. At N=3000N=3000, mean CPU PID time increases from 0.1260.126 seconds at m=5m=5 to 4.2714.271 seconds at m=50m=50.

Comments on the paper

3 comments

(1/3)

We thank the reviewer for their careful assessment, for recognising the strengths of our approach and for providing very helpful remarks. We have done our best to address all the weaknesses and questions raised, as detailed below.

Point 1. Specialist information, removal and reweighting (W1/Q1).

We agree that a removal-based score asks whether the trained model performs better with or without a client, while FedPID measures the information in that client's predictions relative to its peers. To address the reviewer's request, we performed additional experiments in the CIFAR-10 specialist setting, where each client has 5,000 training samples but C1 holds only samples from classes 0–1 and C2–C5 hold samples from the other classes 2–9. We compare FedAvg with all five clients (A), FedAvg excluding C1 throughout training with the other clients' datasets unchanged (B), and FedPID$_R$ with all five clients using U+0.05R+0.05SU+0.05R+0.05S (C). Note that in C, C1 receives a larger aggregation weight than the other clients. Results are reported below as mean accuracy ±\pm sample SD over three seeds.

AccuracyA: FedAvg C1–C5B: Remove C1C: FedPID$_R$
Overall69.83±0.4569.83\pm0.4571.60±0.4271.60\pm0.4276.86±1.0276.86\pm1.02
0: airplane0.00±0.000.00\pm0.000.00±0.000.00\pm0.0062.71±7.7662.71\pm7.76
1: automobile0.00±0.000.00\pm0.000.00±0.000.00\pm0.0053.90±12.9053.90\pm12.90
2: bird81.14±6.9281.14\pm6.9287.43±0.1487.43\pm0.1475.05±2.2775.05\pm2.27
3: cat86.29±2.8986.29\pm2.8977.43±2.2577.43\pm2.2572.76±8.5672.76\pm8.56
4: deer90.86±1.0890.86\pm1.0889.81±0.8289.81\pm0.8286.29±1.3686.29\pm1.36
5: dog74.14±0.5274.14\pm0.5282.00±0.8982.00\pm0.8973.52±4.9473.52\pm4.94
6: frog87.10±2.8987.10\pm2.8992.43±0.5292.43\pm0.5287.48±2.3287.48\pm2.32
7: horse87.52±1.2087.52\pm1.2092.62±0.8692.62\pm0.8680.24±3.6780.24\pm3.67
8: ship94.57±0.8794.57\pm0.8796.38±0.7996.38\pm0.7987.57±3.7187.57\pm3.71
9: truck96.71±0.3896.71\pm0.3897.86±0.3897.86\pm0.3889.05±1.9489.05\pm1.94

In A, FedAvg assigns equal aggregation weights to all clients, so C1's specialist signal is diluted and effectively lost during parameter update averaging. In B, removing C1 improves FedAvg's overall accuracy from 69.8369.83\\% to 71.6071.60\\%, but both specialist classes clearly remain at zero accuracy. Instead, in C, with all five clients retained, FedPID weighting increases C1's mean weight over training, relative to A, from 0.200.20 to 0.4020.402–0.4310.431 across seeds. This improves accuracy on both specialist classes and results in an overall accuracy of 76.8676.86\\%, while mean accuracy on classes 2–9 decreases. C1's specialist information can therefore benefit the model under the tested weighting even though removing C1 improves FedAvg. We believe this experiment sheds further light on how FedPID compares to conventional performance-based methods, and we will add this comparison to Section 5.2.

(2/3)

Point 2. Reliability of raw information estimates (W2/Q2).

We agree that stable rankings alone do not establish accurate component estimates. To address the validation-size check in Q2, we evaluate C1 in the IID, quantity-skew and specialist settings, and both C1 (clean) and C2 (lowest noise level among the noisy clients) in the feature-and-label-noise setting. We use validation subsets of 250, 500, 1,000 and 3,000 examples, keeping the final-round models and PID settings fixed within each of three seeds. Of note, C1 is the specialist in the third setting. The table reports class-prior-weighted information in bits as mean ±\pm sample SD, with total joint MI as Ijoint=U+Uothers+R+SI_{\mathrm{joint}}=U+U_{\mathrm{others}}+R+S.

SettingNNUU (bits)RR (bits)SS (bits)IjointI_{\mathrm{joint}} (bits)
IID2500.000±0.0000.000\pm0.0000.405±0.0100.405\pm0.0100.002±0.0040.002\pm0.0040.434±0.0070.434\pm0.007
5000.000±0.0000.000\pm0.0000.389±0.0090.389\pm0.0090.005±0.0010.005\pm0.0010.430±0.0070.430\pm0.007
1,0000.000±0.0000.000\pm0.0000.385±0.0030.385\pm0.0030.006±0.0020.006\pm0.0020.425±0.0020.425\pm0.002
3,0000.000±0.0000.000\pm0.0000.380±0.0040.380\pm0.0040.005±0.0010.005\pm0.0010.420±0.0010.420\pm0.001
Quantity skew2500.001±0.0010.001\pm0.0010.415±0.0090.415\pm0.0090.003±0.0050.003\pm0.0050.438±0.0040.438\pm0.004
5000.000±0.0000.000\pm0.0000.397±0.0090.397\pm0.0090.002±0.0040.002\pm0.0040.429±0.0070.429\pm0.007
1,0000.000±0.0000.000\pm0.0000.397±0.0060.397\pm0.0060.004±0.0010.004\pm0.0010.431±0.0040.431\pm0.004
3,0000.000±0.0000.000\pm0.0000.386±0.0060.386\pm0.0060.006±0.0000.006\pm0.0000.421±0.0040.421\pm0.004
Specialist2500.027±0.0030.027\pm0.0030.000±0.0000.000\pm0.0000.000±0.0000.000\pm0.0000.339±0.0180.339\pm0.018
5000.028±0.0020.028\pm0.0020.000±0.0000.000\pm0.0000.000±0.0000.000\pm0.0000.324±0.0040.324\pm0.004
1,0000.026±0.0020.026\pm0.0020.000±0.0000.000\pm0.0000.000±0.0000.000\pm0.0000.319±0.0030.319\pm0.003
3,0000.026±0.0010.026\pm0.0010.000±0.0000.000\pm0.0000.000±0.0000.000\pm0.0000.308±0.0030.308\pm0.003
Feature/label noise C1 (clean)2500.053±0.0130.053\pm0.0130.272±0.0200.272\pm0.0200.066±0.0060.066\pm0.0060.403±0.0220.403\pm0.022
5000.062±0.0030.062\pm0.0030.253±0.0150.253\pm0.0150.066±0.0020.066\pm0.0020.386±0.0170.386\pm0.017
1,0000.078±0.0120.078\pm0.0120.230±0.0030.230\pm0.0030.061±0.0010.061\pm0.0010.370±0.0100.370\pm0.010
3,0000.104±0.0040.104\pm0.0040.200±0.0020.200\pm0.0020.045±0.0020.045\pm0.0020.349±0.0010.349\pm0.001
Feature/label noise C2 (noisy)2500.000±0.0000.000\pm0.0000.177±0.0230.177\pm0.0230.023±0.0050.023\pm0.0050.403±0.0220.403\pm0.022
5000.000±0.0000.000\pm0.0000.174±0.0140.174\pm0.0140.023±0.0030.023\pm0.0030.386±0.0170.386\pm0.017
1,0000.000±0.0000.000\pm0.0000.168±0.0060.168\pm0.0060.024±0.0010.024\pm0.0010.370±0.0100.370\pm0.010
3,0000.000±0.0000.000\pm0.0000.162±0.0040.162\pm0.0040.022±0.0010.022\pm0.0010.349±0.0010.349\pm0.001

In the IID, quantity-skew and specialist settings, the raw PID components largely settle as validation-set size increases, with very small changes from 500 examples onwards. In the IID and quantity-skew settings, redundancy remains dominant and changes by less than 33\\% between 500 and 3,000 examples, while unique information remains near zero and synergy remains small. The specialist's unique information stays around 0.0260.026–0.0280.028 bits across all four sizes, with zero redundancy and synergy. Total joint MI also changes only slightly between 500 and 3,000 examples in these three settings. In the feature-and-label-noise setting, C2's unique information remains zero, synergy stays around 0.0220.022–0.0240.024 bits, and redundancy decreases only slightly. C1’s components show some variation across validation sizes, suggesting that more validation examples may be needed for these estimates to settle. Nevertheless, the clean client C1 consistently exhibits higher unique, redundant and synergistic information than noisy clients at every validation size. The full results for all five clients will be included in Appendix D.3.2 alongside Table 12.

(3/3)

Point 3. Tuning FedPID for aggregation and comparing soft and hard predictions (W3).

FedPID's main contribution is to evaluate clients by separating information unique to a client, shared with peers, or available only jointly. Combining these components into a scalar score is optional and is mostly done in this paper to compare our approach to state-of-the-art methods. Choosing aggregation coefficients is then a separate downstream task. MI (U+RU+R) and CMI (U+SU+S) do not separate all three components. The optimal strategy is to choose aggregation coefficients (α,β,γ)(\alpha, \beta, \gamma) by tuning them using validation accuracy, along the lines of what we reported in Table 18. This hyperparameter tuning step includes MI and CMI among the possible choices. We will clarify the decomposition's interpretability purpose and this setting dependence of aggregation performance in Sections 4 and 5.3 and Appendix D.5.

Appendix A.5 changes both the prediction sources and the target treatment. We provide an additional comparison that keeps the first-round specialist models, 3,000 balanced validation examples, binary class targets 1[Y=c]\mathbf{1}[Y=c] and the PID calculation fixed within each of three seeds. Only the sources change: binned class probabilities with edges (0.1,0.4,0.8)(0.1,0.4,0.8) versus full categorical argmax labels, for both C1 and its peers. The table below reports mean ±\pm sample SD in bits for C1's specialist classes. Soft sources give positive unique information on both classes in every seed; hard sources give zero unique information and positive redundancy and synergy. This supports using confidence values to identify C1's specialist information. We will add this comparison to Appendix A.5 and refer to it in Section 4.1.

ClassSourceUnique (UU)Redundant (RR)Synergistic (SS)
0: airplaneSoft0.050±0.0090.050\pm0.0090.000±0.0000.000\pm0.0000.000±0.0000.000\pm0.000
Hard0.000±0.0000.000\pm0.0000.033±0.0080.033\pm0.0080.024±0.0070.024\pm0.007
1: automobileSoft0.160±0.0220.160\pm0.0220.000±0.0000.000\pm0.0000.000±0.0000.000\pm0.000
Hard0.000±0.0000.000\pm0.0000.122±0.0150.122\pm0.0150.047±0.0020.047\pm0.002

Point 4. Participation frequency, historical scores and late entry (Q3).

For FedPID$_F$ to handle clients absent from the first global federation round, we will specify an extension of the first-round schedule in Section 4.4 (p. 6): each client's first round of participation triggers its PID measurement after local training, relative to that round's peers, and its score is then fixed. Previously measured clients retain their scores. Under full participation, this reduces to the submitted first-round schedule.

Under FedPID$_R$, accumulating scores can favour clients with more measured participations: a participating client adds its current score to its stored total AkA_k, while a client that skips the round leaves this total unchanged (Section 4.3; Appendix C, Eqs. (24)–(25)). Consequently, a client that participates more often can receive a larger aggregation weight even when its score per participation is the same as another client’s. Equation (26) instead reports a separate diagnostic: the average of current-round normalised scores over the rounds TkT_k in which client kk participates,

\qquad \bar q_k=\frac{1}{|T_k|}\sum_{t\in T_k}q_k^{(t)}.$$ Absent rounds, including those before a client joins, are excluded. More participations do not by themselves increase this average: with ten equal contributors per round, $\bar q_k=0.1$, even in a pool of 20 or 50 clients. We will clarify the distinction between $A_k$ and $\bar q_k$, and the effect of participation frequency on these stored totals, alongside Eq. (26) in Appendix D.3.1.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.