acceptodds
Under review as a conference paper at ICLR 2027

FedORS: Sample-Efficient Personalized Federated Learning via Orthogonal Residual Sparse Components

Abstract

Personalized federated learning (PFL) often combines a shared representation with client-specific prediction heads. However, these heads cannot recover predictive information absent from the shared representation, limiting performance when clients exhibit heterogeneous structures. We propose FedORS, which augments this architecture with client-specific residual components to capture information beyond the shared representation. In its linear formulation, these components are modeled as sparse and orthogonal to the shared representation subspace, enabling local corrections without increasing the dimension of the shared representation. Under structural and sparsity assumptions, we establish theoretical guarantees on convergence and sample complexity, and further characterize adaptation to previously unseen clients with limited local data. Our analysis highlights how low-dimensional shared structure and residual sparsity can support sample-efficient PFL. Experiments on synthetic and real-world datasets demonstrate improved performance across heterogeneous client distributions and effective adaptation to unseen clients.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Personalized federated learning, Sample efficiency, Sparse residual, Data heterogeneity

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Z2rQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes FedORS. To address the issue where shared representations and personalized prediction heads cannot recover missing information in the shared representations, it introduces an additional client-specific residual branch to improve the accuracy of PFL. Experimental results demonstrate that FedORS outperforms the baselines compared in this study.

Strengths The motivation behind FedORS is relatively interesting, the theoretical analysis is comprehensive, the writing is logically sound, and the experimental results validate the method’s advantages.

Weaknesses

  1. The paper lacks ablation experiments. If such experiments were included, they would provide a more intuitive way to observe the effective gains of each mechanism.
  2. FedORS does not compare itself to the 2026 SOTA PFL methods, which significantly weakens its validity. Additionally, the experimental details provided in the current appendix are insufficient; we hope the authors will provide more detailed experimental information.
  3. The authors use LPC for client-side class segmentation in the paper. Why not use Pathological data segmentation? Furthermore, the paper describes differences in client-side prediction. Where exactly are these differences manifested? Hope the authors can provide an explanation.
Critical strengths

The motivation behind FedORS is relatively interesting, the theoretical analysis is comprehensive, the writing is logically sound, and the experimental results validate the method’s advantages.

Critical weaknesses
  1. The paper lacks ablation experiments. If such experiments were included, they would provide a more intuitive way to observe the effective gains of each mechanism.
  2. FedORS does not compare itself to the 2026 SOTA PFL methods, which significantly weakens its validity. Additionally, the experimental details provided in the current appendix are insufficient; we hope the authors will provide more detailed experimental information.
  3. The authors use LPC for client-side class segmentation in the paper. Why not use Pathological data segmentation? Furthermore, the paper describes differences in client-side prediction. Where exactly are these differences manifested? Hope the authors can provide an explanation.
Questions for authors
  1. The paper lacks ablation experiments. If such experiments were included, they would provide a more intuitive way to observe the effective gains of each mechanism.
  2. FedORS does not compare itself to the 2026 SOTA PFL methods, which significantly weakens its validity. Additionally, the experimental details provided in the current appendix are insufficient; we hope the authors will provide more detailed experimental information.
  3. The authors use LPC for client-side class segmentation in the paper. Why not use Pathological data segmentation? Furthermore, the paper describes differences in client-side prediction. Where exactly are these differences manifested? Hope the authors can provide an explanation.
Reviewer_JQiUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the limitations of shared representations in personalized federated learning: when the shared feature extractor fails to preserve the predictive information needed by clients, adjusting only the personalized classification head is insufficient to compensate for this limitation. The authors propose FedORS, which introduces a local residual branch that directly takes the input, alongside the shared representation and personalized classification head, to supplement client-specific predictive information.

Critical strengths

The research problem is clear and practically relevant. The paper distinguishes between “how to use existing shared features” and “how to supplement information missing from shared features.” This perspective helps explain why personalizing only the classification head may be insufficient in some heterogeneous settings and provides a reasonable motivation for improving client-side predictive capabilities.

The theoretical analysis is connected to the method design. The authors attempt to control the learning complexity of local corrections through sparsity and provide conditional guarantees for shared subspace convergence and adaptation to new clients. This goes beyond an empirical design that simply adds a network branch.

Critical weaknesses

A.The analysis of the mechanistic differences from similar methods and the sources of performance improvements is insufficient.

The paper discusses DBE in the introduction and includes FedDBE as a baseline in Tables 1 and 4, but the relevant experiments mainly compare final classification accuracy. DBE already addresses the conflict between client-specific information and shared representations through local bias memory and mean regularization. Therefore, the authors need to further explain how the “loss of client-side predictive information in shared representations” emphasized by FedORS differs from the “representation shift and degradation” studied by DBE, and why DBE’s mechanism is insufficient to address the problem considered in this paper.

B.The sparsity assumption in Assumption 3 is strong, and the conditions under which it holds and remains valid are insufficiently justified.

This assumption requires not only that the true client-specific residual be sparse, but also that the overall correction vector, which includes the estimation error of the current shared model, remain sparse throughout training. Even if the true residual has only a few nonzero dimensions, the shared model error during early training may still be dense. The paper does not explain why general initialization and iterative updates can preserve this condition. Therefore, it remains unclear which practical structures the sample-efficiency conclusions apply to.

Questions for authors

A.Could the authors use direct representation analysis and controlled experiments to clarify which problems FedORS and DBE each address? For example, using the same backbone and personalized classification head, the authors could compare a fixed local bias, a standard input-dependent residual, and a constrained residual, while controlling the parameter count or computational cost. This would help verify whether the performance improvements indeed come from supplementing information beyond the shared representation and identify the independent effects of the orthogonal and sparse designs.

B.Could the authors provide a concrete, nontrivial setting that satisfies Assumption 3 and explain how initialization should be performed and why the iterative process preserves this property? If the overall correction vector is only approximately sparse, or is not sparse in some rounds, how would the recovery error and convergence conclusions change?

Reviewer_A2ArOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes FedORS, which adds a client-specific residual to the shared representation and personalized prediction head. The residual captures some predictive information that the shared representation misses. For the linear models, this paper assumes the sparse and orthogonal residuals, and derives the convergence and sample-complexity guarantees, including adaptation to the new clients. This paper also presents a nonlinear extension, and evaluates the proposed method on both the synthetic and real data.

Critical strengths
  1. The motivation of this paper is easy to follow. Since a personalized head cannot recover information that is absent from the shared representation, it proposes adding a local residual to address this limitation directly.

  2. The linear results relate the sample requirements to the shared dimension and residual sparsity. The studies of adaptation to the new clients is also useful for understanding what can be learned from the limited local data.

Critical weaknesses
  1. The claim of sample-efficiency is supported by varying the local sample sizes in synthetic experiments. While for the real datasets, only the fixed training configurations are evaluated. Additional comparisons across different local data budgets on the real datasets are necessary to assess whether these gains can persist in practice.

  2. A3 assumes that the true residual plus the current shared-model error stays sparse at every client and at every round. This goes beyond the sparsity of the true residual. When should this assumption hold, and could the analysis be extended to the nonlinear models?

  3. The residual branch increases the local model capacity. Thus, it seems that the comparison with FedRep does not by itself show what the sparsity and orthogonality really contribute to. Can the authors conduct ablations that keep the residual capacity fixed, to separate the effects of sparsity and orthogonality from the benefit of adding a residual branch?

Questions for authors

Please see the weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.