acceptodds
Under review as a conference paper at ICLR 2027

When Does Personalization Help in Federated LoRA Fine-Tuning?

Abstract

Low-rank adaptation (LoRA) makes fine-tuning foundation models inexpensive. Pairing LoRA with federated learning lets clients adapt collaboratively without sharing data. The aim is increasingly a model personalized to each client, but how much personalization helps depends on how clients differ in the amount of local data, label skew, feature shift, and adapter rank. Existing federated LoRA methods each handle only some of these four axes. We propose O4A (All for One, One for All), a single aggregation rule for personalized federated LoRA. O4A distills a shared consensus across clients while preserving each client's personalization. Across the four axes on CIFAR-100 and DomainNet, O4A supports clients at different ranks and improves over existing baselines where personalization matters, giving a regime map of when personalization helps in federated LoRA.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Federated Learning, Low-rank Adaptation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_v8VUOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a new server-side aggregation rule which works as follows:

  1. recomputes each client's full update from their LoRA A & B
  2. runs SVD on each client's update
  3. averages clients' column-space projectors
  4. runs eigendecomposition on the averaged column-space projector and take top-k eigenvectors
  5. averages clients' full update and project it onto the top-k eigenvectors
  6. runs SVD on the resulting matrix and obtain its right singular vectors
  7. project each client's update onto the shared subspace (using the obtained left and right basis) as shown in Eq 5
  8. compute each client's residual
  9. average the shared components
  10. add the averaged shared component to each client's residual and run a truncated SVD, keeping the client's top-r components
  11. the kept components are split into LoRA A & B and send back to the client for the next round

The authors evaluated their proposed method (O4A) over 4 types of data and system heterogeneity against 3 types of existing FL baselines, claiming that existing personalized FL (pFL) works cannot aggregate across ranks while rank-aware FL methods do not personalize.

Critical strengths
  1. The paper has a clear taxonomy to organize the literature and what they are comparing against
  2. O4A is a simple and easy plug-and-play method for server-side aggregation
  3. Considerable improvements in performance over the baselines in the author's evaluation setup
Critical weaknesses
  1. it is unclear if O4A is effective in real-world settings: 1a. the authors claim that O4A is effective across all four axes but did not evaluate on joint feature and label shift. In most real-world applications, all axes exist concurrently. 1b. the number of clients is typically orders of magnitude higher in real-world setups, possible implications: 1b (i). since k is clamp to the client with the lowest r, as the number of clients increase under heterogeneous rank, k will be fixed to a low value. 1b (ii). with many diverse clients, fewer directions get high agreement. 1b (iii). the consensus subspace would be different for each round under partial participation.

  2. heterogeneous rank experiments is only compared with non-personalized FL methods. Are FlexLoRA or HetLoRA further finetuned? If not, it is an unfair comparison, comparing O4A to non-personalized FL works. Comparison to FedRoRA (Wang et al, 2026) is missing despite being mentioned in the Related Work.

  3. k is a sensitive hyperparameter. Moreover, in practice, many existing works show that setting k to be fixed in every layer is suboptimal both in FL (Ramesh & Dess, 2026) and [1]) and LoRA rank sparsity works [2].

[1] Lee et al. "FedP²EFT: Federated Learning to Personalize PEFT for Multilingual LLMs.", AAAI 2026

[2] Zhang et al. "AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning", ICLR 2023

  1. standard personalized FL baselines such as Ditto (Li et al, 2021), FedRep (Collins et al, 2021) are absent despite mentioned (they can be trivially applied to LoRAs at least for the homogeneous rank case)

  2. the authors reported results for FedIT and LoRA-FAIR with no standard deviations from the LoRA-FAIR baseline, but is unclear if that number is comparable with their setup. this also contradicts 'All methods are re-implemented in a single pipeline' in their reproducibility statement.

Questions for authors

a) how does O4A perform under more realistic real-world settings with joint feature and label shift, orders of magnitude more clients, and partial participation?

b) why is local_only missing in most of the result tables? are the single global (non-personalized) FL baselines further finetune for personalization?

Reviewer_DTsvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies personalized federated LoRA under four types of heterogeneity: local data volume, label skew, feature shift, and adapter rank. It proposes O4A, a server-side aggregation rule that reconstructs each client's LoRA update ΔWi=BiAi\Delta W_i=B_iA_i, identifies a shared consensus subspace, and decomposes the update into a shared component GiG_i and a client-specific residual PiP_i. The server aggregates the shared components into G=∑ipiGiG=\sum_i p_iG_i and returns G+PiG+P_i to each client after a rank-rir_i truncated singular value decomposition. The method therefore accommodates clients with different adapter ranks while retaining a personalized component.

Experiments use a ViT-B/16 backbone on CIFAR-100 and DomainNet. They vary label skew, local data volume, feature shift, and adapter rank, and compare O4A with local-only training and federated LoRA baselines. The paper reports improvements across several heterogeneous settings and examines how the consensus rank affects performance.

Critical strengths
  1. A unified aggregation framework for heterogeneous LoRA ranks.

    O4A combines a shared consensus subspace with client-specific residuals. By operating on reconstructed LoRA updates, it supports clients with different adapter ranks while maintaining personalized updates.

  2. Empirical evaluation across multiple heterogeneity settings.

    The paper investigates data scarcity, label skew, feature shift, and adapter-rank heterogeneity on CIFAR-100 and DomainNet. The results demonstrate the potential benefits of personalized aggregation under different heterogeneous conditions.

Critical weaknesses
  1. Lack of theoretical guarantees and direct validation for personalization preservation.

    The paper does not provide a convergence analysis for O4A or a theoretical characterization of how projection and rank truncation affect the preservation of client-specific information.

    More importantly, although personalization is a central contribution, the empirical evaluation mainly relies on predictive accuracy. While per-domain results and client-level gains are reported, these measurements do not directly quantify how much personalized information is retained by the proposed aggregation. The ablation studies mainly examine the consensus rank kk, but do not adequately isolate the contribution of the personalized residual PiP_i. Consequently, it remains unclear whether the reported improvements stem from preserving client-specific information or primarily from the shared consensus component.

  2. Insufficient validation of the claimed generality across the four heterogeneity axes.

    The paper emphasizes that O4A handles four forms of heterogeneity within a unified framework, yet most experiments examine these conditions separately. The combined experiment in Section 5.3 considers data scarcity, label skew, and rank heterogeneity, but does not include feature shift. Thus, the experimental evidence does not fully support the broader claim that O4A remains effective when all four heterogeneous conditions occur simultaneously.

  3. Incomplete comparisons with the most relevant baselines under heterogeneous ranks.

    In Table 2, FedPissa, LoRA-FAIR, and other important baselines lack reported results under the reduced rank budget ∑iri=24\sum_i r_i=24. Although the authors explain that these methods are incompatible with heterogeneous adapter ranks, this does not address the limitation of the comparison. In this setting, O4A is mainly compared against rank-aware global aggregation methods, while the strongest personalized alternatives are absent. Without additional comparisons against compatible personalized alternatives or suitably adapted baselines, the reported high accuracy does not establish superiority over the most relevant competing personalized methods.

  4. Limited performance gains over strong baselines and sensitivity to the consensus rank.

    In the homogeneous-rank DomainNet setting (∑iri=48\sum_i r_i=48), O4A achieves 79.64%79.64\% average accuracy, compared with 79.27%79.27\% for FedPissa and 79.14%79.14\% for local-only training. The improvement is relatively small, and O4A achieves the highest accuracy in only three of the six domains.

    Moreover, Appendix A.4 shows substantial sensitivity to the consensus rank kk, which is chosen via setting-specific ablation. These results raise concerns about the significance and robustness of the reported improvements, especially when the consensus rank cannot be tuned independently for each heterogeneous setting.

Questions for authors
  1. Convergence and retained personalization.

    Can the authors provide an analysis of the optimization behavior of repeated O4A aggregation, or directly quantify the client-specific information preserved after the rank-rir_i truncated SVD? How does this depend on the client rank and consensus rank?

  2. Consensus-rank sensitivity.

    Appendix A.4 shows a large performance difference between k=2k=2 and k=4k=4 at strong label skew. How is kk selected without prior knowledge of the test regime, and how robust is the method when a single selection must serve different clients or changing conditions?

  3. Reduced-budget comparisons.

    Can the authors report rank-matched local-only results for the heterogeneous-rank DomainNet and CIFAR-100 settings? This would help distinguish the benefit of the aggregation rule from the ability to operate with different ranks.

    Can they also clarify the comparability of the DomainNet results marked as taken from prior work?

  4. Scope and computational overhead.

    Can the authors clarify the scope of the claim that O4A handles all four axes together, given that the combined CIFAR-100 experiment does not include feature shift? Can they report the server-side runtime overhead of the dense SVD used in each aggregation round?

Reviewer_Ps5COpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the challenge of reconciling personalization with heterogeneous adapter ranks in federated LoRA. The authors propose O4A, a method that enables personalized aggregation under heterogeneous ranks by reconstructing client-side low-rank updates, extracting a shared consensus subspace, and combining shared components with client-specific residuals. Experiments conducted on CIFAR-100 and DomainNet evaluate the approach across four dimensions, i.e., data scarcity, label skew, feature shift, and rank heterogeneity, which demonstrates the effectiveness under heterogeneous settings.

Critical strengths
  1. The paper proposes O4A, which enables personalized aggregation under heterogeneous ranks by reconstructing low-rank updates and decoupling shared consensus from personalized residuals.
  2. The paper systematically analyzes four types of heterogeneity and, through experiments on visual benchmarks and synthetic datasets, validates both the effectiveness of the proposed method and the noise-reduction mechanism of its shared component.
Critical weaknesses
  1. O4A relies on dense SVD, which entails significant computational overhead on the server side. However, the paper lacks an evaluation of runtime and memory usage for matrices of different sizes, leaving its scalability to larger models and federated settings unclear.
  2. Although truncated SVD yields the optimal low-rank approximation under the Frobenius norm, it does not guarantee the preservation of personalized residuals. The paper lacks an analysis of the extent to which personalized information is retained after truncation, as well as how this retention relates to client-side performance.
Questions for authors
  1. In the heterogeneous-rank experiments, baselines based on shared-factor personalization cannot be applied, while those that support heterogeneous ranks return only a global model. Could the authors include a “local-only” baseline, using the same client-rank configurations and total rank budget, and clarify whether the uniform-rank baseline satisfies the resource constraints of clients with low rank budgets?
  2. The optimal consensus rank k must be determined manually via ablation studies based on data heterogeneity. Could the authors explore a label-agnostic criterion for selecting the consensus rank and verify its stability across various heterogeneity settings?
  3. Section 4.3 claims that k ≤ min_i r_i ensures that each client retains a nonzero personalized residual. However, this constraint does not guarantee that P_i ≠ 0. Furthermore, the subsequent truncated SVD may lead to a loss of personalized information. Could the authors clarify what this constraint actually guarantees, and quantify the extent to which the residual is preserved after truncation, as well as how this relates to client-side performance?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.