- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper studies Byzantine-robust federated learning under heterogeneous client distributions. It proposes a representation-learning framework in which clients maintain personalized linear heads while collaboratively learning a shared nonlinear representation through Byzantine-robust aggregation. The theoretical analysis is detailed and well-motivated for its method. Experiments on multiple datasets show its improvement in byzantine-robustness.
Critical strengths
This paper focus on a practical problem: byzantine-robust federated representation learning with heterogeneous, personalized head.
This paper provides structured theoretical analysis, motivating the proposed method mathematically.
Experiments include regression, multiclass classification, demonstrating the generalization of proposed method.
Critical weaknesses
The experiments compare against only one baseline in sec. 5, resulting insufficient comparison. The reference list includes personalized representation learning [Collins et al., Exploiting Shared Representations for Personalized Federated Learning, icml21] and a Byzantine-robust FedRep framework [Yang et al., FedREP: A Byzantine-Robust, Communication-Efficient and Privacy-Preserving Framework for Federated Learning, 2023], but neither is included in the experimental comparison.
No ablation is reported for double-batch debiasing in sec. 5 and its detailed is unclear. Dose the double-batch resample each round or sample and fix them during the whole training?
Questions for authors
Could the authors clarify how the proposed framework differs from these approaches and whether existing results permit a comparison against a personalized reference using matched aggregation and attack settings? Such evidence would help isolate the contribution beyond introducing personalization.
How does the double-batch debiasing affect the final performance? Dose the double-batch resample each round or sample and fix them during the whole training?
4 comments
Dear Reviewer,
We would like to follow up on our responses to your concerns and questions and kindly ask you to adjust your score if your concerns/question have been addressed.
Dear Reviewer,
We thank you for your time reviewing our work. We are happy to discuss any additional question you may have. If our responses have addressed your questions and concerns, we would appreciate if the score can be adjusted accordingly. Thank you!
Thank you for the response and additional clarification. The response has effectively addressed some of my concerns. I appreciate the rigorous and detailed analysis in the paper. However, the limited validation remains unaddressed. Therefore, I maintain my original rating.
Point-by-point response to Reviewer uECe
We thank the Reviewer for recognizing the practical relevance, structured theoretical analysis, and evaluation on both regression and multiclass classification of our paper.
On the baseline and the distinction between FedRep and FedREP (weakness 1). Section 5 compares a personalized representation-learning method against the standard recipe in Byzantine-robust FL, i.e., learn a common model across heterogeneous clients while handling adversarial updates through robust aggregation. It is already known that such recipe suffers from the unavoidable heterogeneity bias in the non-asymptotics guarantees. The baseline is therefore already Byzantine-robust. Collins et al.'s FedRep (2021) is the alternating personalized representation-learning template on which our method builds. The proposed method can be viewed as a robust version of that template with independent head/representation batches. Yang et al.'s FedREP (2023) is a variant of the standard recipe covered in our baseline which uses a common-model for compression and secure robust aggregation.
On the contribution beyond personalization (question 1). Our contribution is the finite-sample analysis of nonlinear representation learning under adversarial aggregation. Lemma 1, Theorem 1, and Corollary 1 establish heterogeneity, convergence, and recovery bounds without an irreducible model-heterogeneity bias. The experiments illustrate this benefit against the standard robust common-model baseline, with aggregation rules and attacks held fixed.
On what double-batch debiasing does (weakness 2 and question 2). Its role is to separate the label noise used to fit a head from the label noise in its representation gradient. With Ai(t) denoting the head-fitting batch and Bi(t) the independent representation batch, the fitted head is measurable with respect to Ai(t) and the past. For a representation-batch observation, conditional mean-zero label noise, we have
J_{B^{(t)}}(X)^\top h_i^{(t+1)\top}V \mid\mathcal F_t,\mathcal A_i^{(t)},X\right]=0.$$ This is the conditioning used after Eq. (14) in Appendix E. Sample splitting allows us to obtain the stated concentration argument. It does not eliminate the head-estimation error, which Lemma 17 characterizes explicitly. **On resampling each round versus fixing two batches (weakness 2 and question 2).** The theoretical sampling protocol is stated in Section 2.1, under *Data*: each round draws two independent batches of $\tau$ samples, independent of the prior history and across honest clients. Thus, the theory uses fresh batches at every round, rather than fixing two batches once for all of training. Appendix I describes the finite-dataset implementation: a client updates its head and computes its representation update on a separate mini-batch. Double batching separates head-fitting noise from the representation-gradient estimate, enabling the conditional centering used in the finite-sample proof. This is already well established in the literature. Its empirical benefit is not the claim being evaluated. We reemphasize that our central contribution is to characterize robust aggregation under personalized representation learning, with convergence and recovery guarantees that account explicitly for finite-sample errors.