- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper addresses client reliability misjudgments caused by data heterogeneity and participation bias arising from historical trust assessments in Byzantine-robust federated learning. It proposes BRFL-TS, an adaptive client selection method based on Thompson Sampling. The method maintains Beta trust distributions for clients to construct candidate client groups, evaluates group utility using the local evaluation results of candidate aggregated models, and assigns rewards and penalties based on client membership across groups with different utility levels. Client trust and subsequent selection probabilities are dynamically updated, while model reevaluation and rollback mechanisms are introduced to limit the impact of low-utility updates. Theoretically, the paper establishes a lower bound on the probability of receiving renewed trust feedback within finite training windows and an upper bound on the expected waiting time under continued training. Experiments across four datasets and eight untargeted poisoning attacks evaluate the robustness of BRFL-TS under varying data heterogeneity, Byzantine client ratios, and DP, HE, and MPC update protection settings. Client-level decision metrics and trust evolution analyses further demonstrate the effectiveness of the proposed method.
Critical strengths
- Utility-based reliability assessment reduces dependence on individual model-update statistics.
- The adaptive credit assignment mechanism supports reliability reassessment without permanently excluding penalized clients.
- The experimental evaluation provides evidence of robustness and correction of early reliability misjudgments.
Critical weaknesses
- BRFL-TS incorporates multiple components, including Thompson Sampling, adaptive rewards and penalties, cross-group membership ranking, and model rollback. However, it lacks systematic ablation studies, making it difficult to determine each component's independent contribution to overall performance. In particular, direct comparisons between the adaptive reward-and-penalty mechanism and simpler strategies are missing.
- Group-level model utility is used to assess client reliability, but group performance does not directly reflect individual contributions. Under Non-IID settings, differences in data distributions and interactions among clients further complicate credit assignment. The existing experiments lack direct validation of the attribution accuracy of this mechanism.
- BRFL-TS requires constructing multiple candidate models in each training round and having all clients evaluate them. The reported upper bound on plaintext-equivalent communication overhead is approximately 6.5 times that of a full-participation baseline. However, empirical measurements of actual communication volume, computational costs, and runtime are not provided.
- The existing experiments primarily report model accuracy under DP, HE, and MPC settings, but a comprehensive evaluation of the end-to-end privacy-preserving protocol is lacking. The paper has not sufficiently analyzed privacy guarantees, runtime efficiency, and system scalability in practical deployment.
Questions for authors
- Could the authors provide ablation studies of the key components to clarify their individual contributions? What are the respective effects of these components on model accuracy and client-level misclassification rates?
- Does the client-level credit assignment mechanism remain effective under more severe data heterogeneity? There is no direct correspondence between group-level model utility and individual client contributions, and Non-IID data distributions and interactions among clients may affect credit assignment. Could the authors further evaluate the accuracy of reward-and-penalty decisions under different levels of data heterogeneity and examine the relationship between group utility and individual contributions?
- Limited theoretical support for correcting early client misclassification. Theorem 1 establishes a positive lower bound on the probability that a client receives renewed trust feedback within a finite training window. However, this result does not directly guarantee that an initially misclassified benign client will recover its trust. Could the authors further clarify how this theoretical result supports the claim of mitigating the Matthew effect in client participation?
- Could the authors further clarify the end-to-end privacy guarantees under DP, HE, and MPC settings? The existing results primarily demonstrate model accuracy under different privacy protection mechanisms, but the privacy guarantees of the complete protocol have not been sufficiently analyzed. Could the distribution of candidate models and the reporting of client evaluation scores expose additional information? How is such information protected under different privacy-preserving settings?
7 comments
Response to Question 2: Supplementary Information Regarding the Experiment on the Effectiveness of the Credit Allocation Mechanism
We evaluated client-level credit assignment in BRFL-TS under the PoisonedFL attack to assess its effectiveness under stronger data heterogeneity. We used AlexNet on CIFAR-10 with two non-IID Dirichlet partitions ( and ). Each run involved 16 benign and 4 Byzantine clients over 100 training rounds. We focused on benign clients whose first penalty occurred in rounds 1–10.
Decision coverage
To assess how well the credit assignment mechanism revisits early penalties on benign clients, we measured the proportion of these clients that received at least one reward within 10, 25, and 50 rounds of their first penalty. As shown in Table 1, all such clients at received a reward within 10 rounds. Within 25 rounds, every client in the cohort had received a reward, including the client at . These results show that the mechanism can promptly revise its trust assessments in response to new model utility evidence. To further assess this revision, Table 2 reports the clients' final ratios. The posterior mean trust increased markedly for all four clients. These clients also accumulated penalties after round 10. Under heterogeneous data, their updates may perform poorly in some rounds, leaving them eligible for another penalty if they appear in the lowest-ranked group. This behavior is consistent with the design of the decision rule.
Table 1. Coverage of subsequent rewards among benign clients penalized early. Each cumulative proportion uses the full early-penalty cohort in its row as the denominator.
\beta & \leq 10 & \leq 25 & \leq 50 & \text{By round 100} \cr 0.5 & 0/1 & 1/1 & 1/1 & 1/1 \cr 1.0 & 3/3 & 3/3 & 3/3 & 3/3 \cr \text{Pooled} & 3/4 & 4/4 & 4/4 & 4/4 \end{array}$$ #### Trust and recovery-time distribution We measured the waiting time from each client's first erroneous penalty to its first subsequent reward, defined as the first subsequent round in which its reward counter $B$ was incremented. A client's contribution was considered retained when its update was included in an aggregate that passed the rollback check and became the global model used for subsequent training. Both runs had zero rollbacks, so each client's first subsequent reward coincided with its first retained contribution after the penalty. Table 1 reports the cumulative results, and Table 2 gives the recovery timeline for each client. The median waiting time was 7 rounds, with a range of 4–17 rounds. The last recovery occurred at round 26. Table 2 compares each early-penalized client's cumulative reward-to-penalty ratio $B/F$ at the end of rounds 10 and 100. Under the $\mathrm{Beta}(B,F)$ distribution, a higher $B/F$ corresponds to a higher posterior mean trust $B/(B+F)$. This ratio increased markedly for all four clients. At round 100, the sole early-penalized benign client at $\beta=0.5$ had a lower $B/F$ ratio than the three clients at $\beta=1.0$. This observation is consistent with the possibility that stronger heterogeneity can make some benign updates less useful in certain group compositions, making those clients more likely to appear in the lowest-ranked group and receive penalties that lower their posterior mean trust. These results suggest that the credit assignment mechanism can use group utility to revise client-level trust even under heterogeneous data. **Table 2.** Individual recovery times and paired $B/F$ ratios for the early-penalty cohort. Recovery times are in rounds. Each fraction shows the original reward and penalty counters. $$\small\begin{array}{crrrrrr} \beta & \begin{array}{c}\text{Client}\cr \text{ID}\end{array} & \begin{array}{c}\text{First}\cr \text{penalty}\end{array} & \begin{array}{c}\text{Next}\cr \text{reward}\end{array} & \text{Wait} & \begin{array}{c}B/F\cr \text{Round 10}\end{array} & \begin{array}{c}B/F\cr \text{Round 100}\end{array} \cr 0.5 & 12 & 9 & 26 & 17 & 1/2 & 8/6 \cr 1.0 & 4 & 7 & 11 & 4 & 1/2 & 9/6 \cr 1.0 & 18 & 1 & 11 & 10 & 1/2 & 14/4 \cr 1.0 & 19 & 2 & 6 & 4 & 4/2 & 14/6 \end{array}$$ #### False rejection rate (FRR), false acceptance rate (FAR), and final test accuracy Table 3 reports the FRR, FAR, and final CIFAR-10 test accuracy under PoisonedFL. Both decision error rates varied only slightly across the two non-IID settings. Final test accuracy was 81.94% at $\beta=0.5$ and 83.36% at $\beta=1.0$, a difference of 1.42 percentage points. The similar decision error rates and final test accuracies support the robustness of BRFL-TS across the two tested levels of heterogeneity. **Table 3.** False rejection rate, false acceptance rate, and final CIFAR-10 test accuracy at round 100 under PoisonedFL. All values are percentages. $$\begin{array}{lrrr} \beta & \text{FRR} & \text{FAR} & \text{Final test accuracy} \cr 0.5 & 7.23 & 7.94 & 81.94 \cr 1.0 & 7.55 & 9.09 & 83.36 \end{array}$$Response to Question 3: A Supplement Regarding the Correction of Early Client-Side Misjudgment Theories
Thank you for pointing out the gap between “receiving feedback again” and “recovering from an erroneous early penalty.” Our motivation for this proof is that selection rules that favor clients with “strong historical performance” may repeatedly select the same clients, leaving insufficiently explored clients with few opportunities to build a performance history. We therefore wanted to show that, in BRFL-TS, Thompson sampling still gives a client a chance to enter a candidate group even after poor early feedback. The theoretical bound on the probability of feedback offers limited support for this conclusion. We discussed this issue in depth and believe that the underlying reason is that, among the multiple candidate models in our method, clients in groups with intermediate scores receive no feedback. The proof therefore provides a lower bound on the probability of feedback only for the case in which the client appears in every group. In practice, whether a client receives feedback is closely tied to its actual performance, a relationship that the theoretical analysis did not account for. We therefore drew on the ideas of Good Arm Identification [1] and composite anonymous feedback [2] to formulate a utility separation assumption and derive a new lower bound on the probability of feedback.
First, we derive a lower bound on the probability that a client is selected in draws without replacement in a single round. Given client 's Thompson sample , if the first draws all miss , the total remaining weight of the other clients is at most . Multiplying the upper bounds on the probability of missing at each draw and taking the complement gives the following lower bound on the probability that a group of size contains .
Response to Question 1: Additional information regarding ablation studies
Thank you for your comments on our paper. We conducted an ablation study of BRFL-TS under the PoisonedFL attack using CIFAR-10 and AlexNet. All configurations used a non-IID Dirichlet data partition (β=0.5), with 20 clients, 20% of which were Byzantine, and ran for 100 communication rounds. In each round, we formed 10 candidate groups of 3 clients each. We used the same three random seeds (42, 123, and 3407) for every configuration.
For each run, we report model test accuracy and two client-level error rates. The false rejection rate (FRR) is the proportion of explicit decisions on benign clients that assign a penalty. The false acceptance rate (FAR) is the proportion of explicit decisions on Byzantine clients that assign a reward.
Table 1. Ablation study of BRFL-TS on CIFAR-10 under PoisonedFL. Values are percentages reported as mean ± sample standard deviation across three random seeds.
\text{Configuration} & \text{Model test accuracy} & \text{FRR} & \text{FAR} \cr \text{Full BRFL-TS} & 79.97\pm1.26 & 6.84\pm1.62 & 8.59\pm3.55 \cr \text{Uniform sampling} & 79.46\pm1.32 & 11.32\pm0.98 & 7.69\pm2.42 \cr \text{Fixed penalty quota} & 79.84\pm1.40 & 33.27\pm1.68 & 5.75\pm2.02 \cr \text{Random assignment} & 79.63\pm0.90 & 8.76\pm1.52 & 8.31\pm1.42 \cr \text{No rollback} & 79.94\pm0.82 & 8.44\pm1.87 & 5.55\pm0.90 \end{array}$$ #### Thompson sampling Replacing trust-weighted group sampling with uniform sampling increased FRR by 4.48 percentage points and decreased model test accuracy by 0.51 percentage points. The most notable benefit of Thompson sampling was a lower rate of erroneous penalties on benign clients, reflecting its ability to correct early misclassifications. #### Adaptive penalty quota Fixing the penalty quota at two per round increased FRR by 26.43 percentage points and decreased model test accuracy by 0.13 percentage points. In the full method, the adaptive rule assigned an average of 0.46 penalties per round. This comparison shows that the adaptive quota had the largest effect on the false rejection of benign clients. As client selection progressively narrows and filters out malicious clients, fewer malicious clients remain in the lowest-ranked group. These results demonstrate that the adaptive reward-and-penalty mechanism adjusts the penalty quota in response to this change, thereby reducing false rejection. #### Lexicographic penalty assignment Replacing lexicographic assignment with random selection of penalty recipients from the lowest-ranked group increased FRR by 1.92 percentage points and decreased model test accuracy by 0.34 percentage points. The average penalty quotas were similar (0.44 with random assignment and 0.46 in the full method). This suggests that using membership information from utility-ranked groups helps identify malicious clients to penalize more accurately at a comparable penalty budget, thereby limiting the false rejection of benign clients. #### Rollback Disabling rollback increased FRR by 1.60 percentage points and decreased model test accuracy by about 0.03 percentage points, yielding accuracy close to that of the full method. In the full method, rollback was triggered an average of 0.33 times per run. It also occurred in other ablation configurations. Rollback primarily serves as a safeguard against cases where a malicious update in the highest-ranked group substantially degrades model accuracy. Its effect is less pronounced when the other mechanisms are functioning as intended. Under this configuration, the complete BRFL-TS scheme achieves the lowest average FRR and the highest average model test accuracy. As for why the average FAR remains largely similar across the various ablation variants, this is because the optimal group is selected based on model utility, and robust aggregation—from which the FAR is calculated—is performed using client updates from this optimal group. Since removing any of the aforementioned modules does not fundamentally alter this process, the FAR remains within essentially the same range.