acceptodds
Under review as a conference paper at ICLR 2027

Byzantine-Robust Federated Representation Learning

Abstract

We study federated learning (FL) with adversarial clients, where the goal is to minimize the average loss of the honest (non-adversarial) clients without knowing their identity. Under heterogeneity, a single shared model parameter is statistically inappropriate: it cannot capture the distinct data-generating processes across clients, incurring an irreducible model-heterogeneity bias and severely limiting robustness to adversarial clients (a.k.a. Byzantine-robustness). We address this problem through representation learning, where each client learns a personalized linear head, while collaboratively estimating a shared nonlinear representation through Byzantine-robust aggregation. We demonstrate that the heterogeneity among honest representation gradients is only controlled by the representation error and statistical errors that decay either with the number of data samples per client () or the number of iterations (). In particular, our non-asymptotic parameter recovery bound reveals three terms: (i) an initialization-dependent error that goes away with , (ii) finite-sample noise terms that decreases with and the number of honest clients, and (iii) a stochastic gradient variance term that also reduces with . Importantly, with no irreducible model-heterogeneity bias in our bounds. We extend our regression analysis to multiclass classification, and empirically validate it on CIFAR-10, FEMNIST, and School Exam Score datasets.

Primary area:
learning theory
Keywords:
Byzantine robustness, federated learning, nonlinear representation learning, non-asymptotic analysis

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_uECeOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies Byzantine-robust federated learning under heterogeneous client distributions. It proposes a representation-learning framework in which clients maintain personalized linear heads while collaboratively learning a shared nonlinear representation through Byzantine-robust aggregation. The theoretical analysis is detailed and well-motivated for its method. Experiments on multiple datasets show its improvement in byzantine-robustness.

Critical strengths

This paper focus on a practical problem: byzantine-robust federated representation learning with heterogeneous, personalized head.

This paper provides structured theoretical analysis, motivating the proposed method mathematically.

Experiments include regression, multiclass classification, demonstrating the generalization of proposed method.

Critical weaknesses

The experiments compare against only one baseline in sec. 5, resulting insufficient comparison. The reference list includes personalized representation learning [Collins et al., Exploiting Shared Representations for Personalized Federated Learning, icml21] and a Byzantine-robust FedRep framework [Yang et al., FedREP: A Byzantine-Robust, Communication-Efficient and Privacy-Preserving Framework for Federated Learning, 2023], but neither is included in the experimental comparison.

No ablation is reported for double-batch debiasing in sec. 5 and its detailed is unclear. Dose the double-batch resample each round or sample and fix them during the whole training?

Questions for authors

Could the authors clarify how the proposed framework differs from these approaches and whether existing results permit a comparison against a personalized reference using matched aggregation and attack settings? Such evidence would help isolate the contribution beyond introducing personalization.

How does the double-batch debiasing affect the final performance? Dose the double-batch resample each round or sample and fix them during the whole training?

4 comments
AuthorsOpenReview ↗

Point-by-point response to Reviewer uECe

We thank the Reviewer for recognizing the practical relevance, structured theoretical analysis, and evaluation on both regression and multiclass classification of our paper.

On the baseline and the distinction between FedRep and FedREP (weakness 1). Section 5 compares a personalized representation-learning method against the standard recipe in Byzantine-robust FL, i.e., learn a common model across heterogeneous clients while handling adversarial updates through robust aggregation. It is already known that such recipe suffers from the unavoidable heterogeneity bias in the non-asymptotics guarantees. The baseline is therefore already Byzantine-robust. Collins et al.'s FedRep (2021) is the alternating personalized representation-learning template on which our method builds. The proposed method can be viewed as a robust version of that template with independent head/representation batches. Yang et al.'s FedREP (2023) is a variant of the standard recipe covered in our baseline which uses a common-model for compression and secure robust aggregation.

On the contribution beyond personalization (question 1). Our contribution is the finite-sample analysis of nonlinear representation learning under adversarial aggregation. Lemma 1, Theorem 1, and Corollary 1 establish heterogeneity, convergence, and recovery bounds without an irreducible model-heterogeneity bias. The experiments illustrate this benefit against the standard robust common-model baseline, with aggregation rules and attacks held fixed.

On what double-batch debiasing does (weakness 2 and question 2). Its role is to separate the label noise used to fit a head from the label noise in its representation gradient. With Ai(t)\mathcal A_i^{(t)} denoting the head-fitting batch and Bi(t)\mathcal B_i^{(t)} the independent representation batch, the fitted head is measurable with respect to Ai(t)\mathcal A_i^{(t)} and the past. For a representation-batch observation, conditional mean-zero label noise, we have

J_{B^{(t)}}(X)^\top h_i^{(t+1)\top}V \mid\mathcal F_t,\mathcal A_i^{(t)},X\right]=0.$$ This is the conditioning used after Eq. (14) in Appendix E. Sample splitting allows us to obtain the stated concentration argument. It does not eliminate the head-estimation error, which Lemma 17 characterizes explicitly. **On resampling each round versus fixing two batches (weakness 2 and question 2).** The theoretical sampling protocol is stated in Section 2.1, under *Data*: each round draws two independent batches of $\tau$ samples, independent of the prior history and across honest clients. Thus, the theory uses fresh batches at every round, rather than fixing two batches once for all of training. Appendix I describes the finite-dataset implementation: a client updates its head and computes its representation update on a separate mini-batch. Double batching separates head-fitting noise from the representation-gradient estimate, enabling the conditional centering used in the finite-sample proof. This is already well established in the literature. Its empirical benefit is not the claim being evaluated. We reemphasize that our central contribution is to characterize robust aggregation under personalized representation learning, with convergence and recovery guarantees that account explicitly for finite-sample errors.

Dear Reviewer,

We would like to follow up on our responses to your concerns and questions and kindly ask you to adjust your score if your concerns/question have been addressed.

Dear Reviewer,

We thank you for your time reviewing our work. We are happy to discuss any additional question you may have. If our responses have addressed your questions and concerns, we would appreciate if the score can be adjusted accordingly. Thank you!

Thank you for the response and additional clarification. The response has effectively addressed some of my concerns. I appreciate the rigorous and detailed analysis in the paper. However, the limited validation remains unaddressed. Therefore, I maintain my original rating.

Reviewer_fe7mOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies Byzantine-robust nonlinear representation learning in heterogeneous federated learning. The authors consider a shared nonlinear representation with client-specific linear prediction heads. The method alternates between optimizing the personalized heads and updating the shared representation through Byzantine-robust aggregation of representation-gradient momenta. Theoretically, the authors show that, under a shared representation assumption, the gradient heterogeneity among honest clients can be controlled without an irreducible model-heterogeneity bias term. Based on this result, the paper establishes high-probability convergence guarantees and non-asymptotic parameter recovery bounds, and extends the theoretical analysis to multiclass classification. Experiments evaluate the proposed method under different Byzantine attack settings.

Critical strengths
  1. The paper is well written and easy to follow.
  2. The theoretical analysis is comprehensive, with detailed proofs provided in the appendix.
Critical weaknesses
  1. The proposed method mainly combines existing techniques from personalized federated representation learning and Byzantine-robust gradient momentum aggregation. The algorithmic novelty appears relatively limited, with the main contribution lying in the theoretical analysis.

  2. BR-MTRL [1] has already explored the combination of shared representations, personalized prediction heads, alternating optimization, and Byzantine-robust aggregation. However, this closely related work is neither cited nor discussed, and no experimental comparison is provided. The authors are encouraged to clarify the differences between the two approaches and better highlight the contributions of this paper.

[1] Byzantine Resilient Federated Multi-Task Representation Learning. arXiv preprint arXiv:2503.19209.

  1. The experiments mainly focus on predictive performance, without directly analyzing gradient heterogeneity among honest clients or robust aggregation errors. Additional empirical analysis of these quantities would help better connect the experimental results with the theoretical findings.

  2. The experiments lack direct comparisons with other relevant Byzantine-robust personalized federated learning methods. The limited baseline coverage makes it difficult to fully assess the advantages of the proposed approach over existing methods.

Questions for authors

I would appreciate it if the authors could address the concerns raised above.

5 comments
AuthorsOpenReview ↗

Point-by-point response to Reviewer fe7m

We thank the Reviewer for recognizing the clarity of the paper and the comprehensive theoretical analysis.

On the algorithmic template (weakness 1). The main contribution is indeed theoretical. Alternating head/representation updates and robust momentum aggregation are well established tools as we discuss in the introduction and setup of the paper. Combining them does not by itself imply that honest-gradient heterogeneity has no persistent model-heterogeneity term: the fitted heads change with the representation and introduce finite-sample error, while momentum and robust aggregation couple errors across rounds. Lemma 17 controls head estimation, Lemma 21 transfers curvature to learned heads, Lemma 24 closes the heterogeneity bound, and Appendix F proves the convergence result. Appendix G then establishes the bound on the parameter recovery error.

On BR-MTRL and the missing discussion (weakness 2). We agree that BR-MTRL should be cited and discussed. We will do it in the reviewed version of the paper. However, we emphasize that BR-MTRL does not establish the finite-sample theory analyzed here. Our results apply to the class of (f,κ)(f,\kappa)-robust aggregators under the stated conditions and assumption, with explicit dependence on number of samples per clients, number of honest clients, label noise level, and stochastic-gradient noise. We do not claim a new shared-backbone architecture or an empirical improvement over BR-MTRL.

On measuring gradient heterogeneity and aggregation error (weakness 3). These are the quantities controlled directly in the analysis, i.e., Appendix E, Eqs. (19)--(20), bounds honest-gradient heterogeneity through recovery and sampling errors. Appendix C.4, Lemmas 11--12, bounds the heterogeneity of honest momenta and the resulting aggregation error. In particular, we have

\leq\kappa D_T \lesssim\kappa\bar G_T+ \kappa\frac{1-\beta}{1+\beta}\sigma_{\mathsf g}^2 \left(1+\frac1{|\mathcal H|}\right)\log(T/\delta).$$ This connects the statistical analysis to the effect of adversarial updates. Our experiments measure prediction performance rather than these intermediate quantities. In Figure 1, we vary the number of honest clients with five Byzantine updates fixed, in Tables 1--2 we hold the aggregator and attack fixed when comparing personalization. They provide supporting empirical evidence for our non-asymptotic analysis. **On the experimental baseline (weakness 4).** We emphasize that our baseline follows the standard Byzantine-robust FL formulation: learning a common model across heterogeneous clients using robust aggregation to handle adversarial updates. As described in Section 5 and Appendix I, both methods use the same NNM preprocessing, robust aggregators, backbone, loss, momentum, and attacks. Thus, Tables 1--2 characterize the benefit of personalized heads under matched robust aggregation. This comparison directly supports our central theoretical contribution, i.e., finite-sample convergence and recovery guarantees without an irreducible model-heterogeneity bias. We also refer the reviewer to our response to Reviewer xecL, where we provide additional comparisons.

Dear Reviewer,

We would like to follow up on our responses to your concerns and questions and kindly ask you to adjust your score if your concerns/question have been addressed.

Thanks for the rebuttal. While I appreciate the clarification of the theoretical contributions, my concerns about limited methodological novelty and insufficient empirical validation remain. Hence, I maintain my original rating.

Reviewer_xecLOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the authors study the problem of representation learning across heterogenous clients in Byzantine Federated Learning. They propose a paradigm that each client has a shared backbone and a client-specific linear head: the backbone extracts information shared across the heterogenous clients, and the linear head provides personalized information. Then they propose a gradient-based algorithm to train the head and the backbone simultaneously, and develop convergence for the algorithm. Finally, they conduct experiments to demonstrate the effectiveness of the proposed method.

Critical strengths
  • The convergence analysis is sound and solid.
  • The empirical performance of the proposed method is good.
Critical weaknesses
  • The idea of a shared backbone + personalized head for Byzantine Federated Learning is not novel. It was explored in [1], which is not cited in the paper.
  • The authors motivate the paper by "the server cannot tell the difference between honest heterogeneity and adversarial behavior when the honest clients are heterogeneous" (Line 44), and propose the shared backbone + personalized head method. However, they did not show that the proposed method address the problem (i.e., telling "the difference between honest heterogeneity and adversarial behavior").
  • The experiment results are not comprehensive. The authors do not compare the proposed method with baselines such as FedREP (Yang et al., 2023) or FedPer [2].

[1] Le & Moothedath. Byzantine Resilient Federated Multi-Task Representation Learning. arXiv:2503.19209
[2] Arivazhagan et al. Federated Learning with Personalization Layers. arXiv:1912.00818

Questions for authors
  • Questions
    • How does the proposed method address the problem of "the server cannot tell the difference between honest heterogeneity and adversarial behavior when the honest clients are heterogeneous"?
    • What are the performance of FedREP and FedPer in the same experiment setting?
  • Other issues
    • The format of some references is not correct. For example:
      • Line 136: (Zhang et al., 2024a) -> Zhang et al. (2024a)
      • Line 174: Collins et al. (2021) -> (Collins et al., 2021), Zhang et al. (2024a) -> (Zhang et al., 2024a)
4 comments
AuthorsOpenReview ↗

Point-by-point response to Reviewer xecL

We thank the Reviewer for acknowledging our convergence analysis as sound and solid and the empirical performance as good.

On novelty and BR-MTRL (weakness 1). We agree that the shared-backbone/private-head architecture predates this work. We thank the Reviewer for pointing out BR-MTRL, which will be added to the related work. However, BR-MTRL does not provide the statistical analysis established here: the coupled bounds for honest-gradient heterogeneity, population-gradient convergence, and prediction recovery with noisy finite-sample head fitting and arbitrary adversarial updates.

On distinguishing heterogeneity from adversarial behavior (weakness 2 and question 1). Identifying Byzantine clients is not the task solved by our method, and it not typically the task solved by Byzantine-robust FL. As Definition 1 shows, robustness depends on the heterogeneity of the honest updates, even though their identities are unknown to the server. Our analysis derives a bound on this heterogeneity after personalization. In particular, Appendix E, Eq. (19), provides

+C\sigma^2\frac{p\vee rq+\log(|\mathcal H|/\delta)}{\tau}.$$ Thus, different true heads do not produce a separate, persistent head-heterogeneity term. Lemma 1 and Theorem 1 close the dependence on representation recovery over training. Together with Lemma 12, this controls the aggregation error without requiring the server to label any clients. **On FedREP, FedRep, and FedPer (weakness 3 and question 2).** The similarly named methods address different questions. Yang et al.'s *FedREP* (2023), cited in the review, learns a common model using consensus sparsification, secure aggregation, and robust aggregation. It does not use the personalized representation/head formulation we have here nor provide the statistical analysis we provide. Collins et al.'s *FedRep* (2021) uses that formulation, with guarantees for a non-adversarial linear setting. Our alternating procedure follows that established alternating optimization and characterizes its robust nonlinear statistical results under adversarial and heterogeneous clients. FedPer uses shared base layers and local personalization layers with joint local training and averaging, without any Byzantine robustness guarantee in its formulation. We also emphasize that the reported baseline is exactly what the literature in Byzantine robust FL will do. It learns a common model parameter across heterogeneous clients with aggregation to account for adversarial clients. This is described in Section 5 and Appendix I. Our proposed method is the personalized, FedRep-style robust method method. We have not reported implementations of Yang et al.'s FedREP or FedPer, as they consider different settings as detailed above. We reemphasize that in Table 1, we have our method achieving $87.89$--$88.79$\% on CIFAR-10 and $91.89$--$93.67$\% on FEMNIST across the tested losses, attacks, and aggregators, which substantially outperforms the standard recipe of Byzantine-robust FL (here refereed to as baseline). Below we provide the additional numerical experiment (for the Exam School Data - Regression task) requested by the Reviewer. | Method | Aggregation | Final MSE $\downarrow$ | | --- | --- | --- | | FedPer | Average | $0.7158 \pm 0.0756$ | | FedREP | Buffered TrMean | $0.7778 \pm 0.0357$ | | Ours | NNM+TrMean | $\mathbf{0.6316 \pm 0.0275}$ | As we can see, our approach also outperforms FedPer and FedREP.

Dear Reviewer,

We would like to follow up on our responses to your concerns and questions and kindly ask you to adjust your score if your concerns/question have been addressed. We are also happy to address any additional question about the paper you may have.

Dear Reviewer,

We thank you for your time reviewing our work. We are happy to discuss any additional question you may have. If our responses have addressed your questions and concerns, we would appreciate if the score can be adjusted accordingly. Thank you!

Thank you for your response. The rebuttal addressed part of my concerns. This paper provided some useful results. But given its limited novelty and impact, I decide to keep my original score.

Reviewer_F5WFOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a Byzantine-robust federated learning framework with a shared nonlinear representation and client-specific linear heads. It alternates exact local head fitting and representation-gradient estimation on independent fresh batches, followed by robust aggregation of gradient momenta. Its main contribution is a finite-sample analysis connecting honest-gradient heterogeneity to representation error and statistical noise, yielding convergence and prediction-error guarantees under realizability, local curvature, local iterate containment, and a sufficiently small robustness coefficient, with a classification extension under a margin condition. Experiments on CIFAR-10, FEMNIST, and School Exam Score show improved local prediction over shared-model training under ALIE and Mimic attacks, with personalized FEMNIST training remaining close to its no-attack reference.

Critical strengths

The finite-sample analysis targets an important unresolved part of robust personalization. Earlier work establishes the broader mechanism: Mishchenko et al., Partially Personalized Federated Learning: Breaking the Curse of Data Heterogeneity, Theorem 4 and Appendix D, Theorem 8 give robust convergence under operator assumptions and estimator second moments that vanish with the local operator. The present paper instead seeks to account explicitly for finite-sample head fitting, label noise, and additive gradient noise. Its proposed chain Lemma 1 controlling honest-gradient heterogeneity, Theorem 1 controlling the population gradient, and Corollary 1 controlling prediction error, would show why these sources of error need not recreate a persistent heterogeneity floor.

That is a meaningful extension of Toso et al., On the Gradient Heterogeneity Dynamics of Adversarially Robust Federated Regression, Lemma 4.1 and Theorem 4.1, whose shared-model analysis retains a function-heterogeneity term. Learning the heads changes the gradients throughout training, so the proposed extension requires controlling additional estimation and optimization errors. The fact that robust shared-representation architectures already exist, including Le and Moothedath's Byzantine Resilient Federated Multi-Task Representation Learning, Section III, does not diminish the value of establishing this statistical guarantee.

The experiments support its motivation: Table 1 shows better local classification under both tested attacks and NNM-preprocessed aggregation rules, while Figure 1 shows personalized FEMNIST training remaining close to its own no-attack reference. These observations support the usefulness of the setting; the theoretical bounds carry the main contribution.

Critical weaknesses

The decisive step establishing the new guarantee cannot be checked from the supplied file. Lemma 1, Theorem 1, and Corollary 1 refer to Appendices E-G, and the momentum analysis invokes Lemmas 12-13. The provided twelve-page PDF ends with references and does not contain these arguments. This matters specifically because the new contribution depends on controlling errors from repeatedly fitted heads through robust momentum aggregation and then translating gradient control into prediction recovery. Assumption 3 gives curvature with the true heads held fixed; Section 3 adds a curvature inequality, local iterate containment, and a small-robustness-coefficient condition. The proof is needed to verify that these stated conditions suffice for the claimed chain. Without it, the available evidence establishes the motivation and reported empirical benefits, but leaves the principal theoretical advance unresolved.

Questions for authors
  1. Required verification: Can the referenced proofs be made available, particularly the argument controlling finite-sample head-fitting error through Lemma 1 and the momentum bounds?
  2. Clarification: In the recovery argument, how does the additional Section 3 curvature inequality connect fixed-true-head curvature in Assumption 3 to prediction recovery with refitted heads?
3 comments
AuthorsOpenReview ↗

Point-by-point response to Reviewer F5WF

We thank the Reviewer for acknowledging that our finite-sample analysis is meaningful and that the control of repeatedly fitted heads is an important challenge that we address in our analysis.

Location of the proofs (weakness and question). The arguments are contained in the appendix accompanying the paper (please refer to our supplementary material). The twelve-page main-paper PDF described in the review does not include the supplementary appendix. To make verification direct, the relevant locations are:

ArgumentLocation in the appendix
Overall proof structureAppendix D, Proof Roadmap
Stochastic-gradient and momentum controlAppendix C.4, Lemmas 10--13
Finite-sample head estimationAppendix E, Lemmas 15--17, Eqs. (17)--(18)
Prediction error and gradient heterogeneityAppendix E, Lemma 18, Eqs. (19)--(20)
True-head to learned-head curvatureAppendix E, Lemmas 19--21
Coupled representation and heterogeneity boundsAppendix E, Theorem 3 and Lemma 24
Population-gradient convergence (Theorem 1)Appendix F, Theorem 4
Representation and prediction recovery (Corollary 1)Appendix G.1--G.2, Corollaries 3--4

On how finite-sample head fitting enters the proof. In Equation (18) we have the exact decomposition of the empirical minimizer. By writing ϕt=ϕB(t)\phi_t=\phi_{B^{(t)}} and using the head-fitting batch, we obtain

h_i^{(t+1)}-h_i^\star &=h_i^\star\left[\frac1\tau\sum_k(\phi_{B^\star}(X_{i,k})-\phi_t(X_{i,k}))\phi_t(X_{i,k})^\top\right]\widehat\Sigma_{i,\phi}^{-1}+\left[\frac1\tau\sum_k V_{i,k}\phi_t(X_{i,k})^\top\right]\widehat\Sigma_{i,\phi}^{-1}. \end{aligned}$$ Lemma 15 then controls the inverse covariance after the sample burn-in, and Lemma 17 bounds the two terms by representation error and finite-sample label noise. Lemma 18 then controls prediction error. On the independent representation batch, the signal-noise decomposition in Eqs. (11)--(14) yields $$\bar G^{(t)}\leq C\bar E^{(t)} +C'\sigma^2\frac{p+\log(|\mathcal H|/\delta)}{\tau}.$$ Lemmas 11--12 transfer the gradient heterogeneity to momentum term and robust-aggregation error. Lemma 13 controls momentum the momentum term. This is why finite-sample errors are kept in the final result rather than assumed to vanish, as we consider a finite number of communication rounds and a finite number of data samples per client. **On fixed-head curvature versus recovery with refitted heads (clarification question).** Assumption 3 alone provides curvature with the true heads held fixed. The additional inequality in Section 3 is used to obtain positive curvature after replacing them with learned heads. In particular, Lemma 21 decomposes the empirical gradient into the population gradient with true heads, the learned-head term, and an empirical-process error. By writing $d=B-B^\star$, the proof provides the following expression $$\begin{aligned} \langle\nabla_B L_{\mathcal H}(\{h_i^{(t+1)}\},B),d\rangle &\geq\left(\frac{3\mu_3}{4}-\frac{2L_3H\bar J\bar\phi}{\mu_1}\right)||d||^2 -C_{\rm rep}p_\sigma\mathfrak s_{\mathcal H,\tau} -C_{\rm head}\sigma^2\mathfrak s_\tau\geq\frac{\mu_3}{4}||d||^2 -C_{\rm rep}p_\sigma\mathfrak s_{\mathcal H,\tau} -C_{\rm head}\sigma^2\mathfrak s_\tau, \end{aligned}$$ where the second inequality uses $\mu_3\geq4L_3H\bar J\bar\phi/\mu_1$. Here the statistical rates include the union bound over clients and communication rounds. This estimate is applied at the current iterate, where the heads are fitted. Theorem 3 then combines it with the update recursion. Appendix G converts the resulting representation bound into prediction recovery using Lemma 18. The guarantee is local as stated in Section 3, which is standard for nonlinear representation learning.

Dear Reviewer,

We would like to follow up on our responses to your concerns and questions and kindly ask you to adjust your score if your concerns/question have been addressed. We are also happy to address any additional question about the paper you may have.

Dear Reviewer,

We thank you for your time reviewing our work. We are happy to discuss any additional question you may have. If our responses have addressed your questions and concerns, we would appreciate if the score can be adjusted accordingly. Thank you!

Comments on the paper

1 comment
AuthorsOpenReview ↗

General comment to the Reviewers

First of all, we thank the Reviewers for their constructive comments and feedback. We appreciate that they recognize the importance of personalization for adversarially robust FL, our theoretical contributions, and the empirical performance. Below, we first clarify two points common to the reviews.

Our contribution is a finite-sample guarantee for robust nonlinear representation learning. Shared representations, personalized heads, and alternating updates are established ideas (see the Introduction and Appendix B of the supplementary material). Existing analyses leave open how repeatedly fitting personalized heads from noisy, finite samples affects robust aggregation of representation-gradient momenta. This matters because head estimation error can increase the heterogeneity among honest updates, making it harder for the robust aggregator to estimate their mean. Lemma 1 bounds this heterogeneity in terms of representation and statistical errors, Theorem 1 controls the population representation gradient, and Corollary 1 bounds prediction recovery error. We note that this is the first rigorous non-asymptotic analysis for adversarially robust federated representation learning presented in the literature.

Let us also position our results with respect to BR-MTRL (Le and Moothedath, 2025). This is indeed a relevant prior work and we will cite it properly in the revised version. This work studies a closely related architecture and provides empirical evidence, but does not establish the finite-sample convergence and recovery guarantees studied here. In addition, Mishchenko et al. (2023) (already discussed in the submission), do provide some asymptotic analysis (non finite-sample analysis) robust personalization theory. Our analysis explicitly characterizes the non-asymptotic guarantees for nonlinear representation learning for adversarially robust FL.

Our experiments compare against the robust common-model baseline. Both methods use the same backbone, loss, mini-batch size, momentum, aggregation rule, and attack. In Appendix I we specify the training and evaluation protocol. The baseline aggregates the full common model, while our method keeps private heads and aggregates only representation updates. In particular, the baseline already uses NNM+Krum or NNM+trimmed mean, following exactly the Byzantine-robust optimization literature. The benefit of importance of replacing the standard aggregation averaging under arbitrary adversarial updates is established by Blanchard et al. (2017), Yin et al. (2018), Allouah et al. (2023), and many other works. Our comparison instead asks how personalization can help in adversarially robust learning under heterogeneous clients.

The evaluation includes two attacks, two robust aggregators, three datasets, two classification losses, and a sweep over the number of honest clients. Tables 1--2 and Figure 1 report these results. Appendix I provides the implementation details. Our experiments support the theoretical results. The idea is to not rank our results against every personalization method. There is clear baseline which is adversarially robust learning with a common model parameter across heterogeneous clients. We compare our representation learning method against that method. It is also important to distinguish Collins et al.'s FedRep (2021), a personalized representation-learning method, from Yang et al.'s FedREP (2023), an approach for compression and secure aggregation, and Byzantine robustness of a common model.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.