- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes a new server-side aggregation rule which works as follows:
- recomputes each client's full update from their LoRA A & B
- runs SVD on each client's update
- averages clients' column-space projectors
- runs eigendecomposition on the averaged column-space projector and take top-k eigenvectors
- averages clients' full update and project it onto the top-k eigenvectors
- runs SVD on the resulting matrix and obtain its right singular vectors
- project each client's update onto the shared subspace (using the obtained left and right basis) as shown in Eq 5
- compute each client's residual
- average the shared components
- add the averaged shared component to each client's residual and run a truncated SVD, keeping the client's top-r components
- the kept components are split into LoRA A & B and send back to the client for the next round
The authors evaluated their proposed method (O4A) over 4 types of data and system heterogeneity against 3 types of existing FL baselines, claiming that existing personalized FL (pFL) works cannot aggregate across ranks while rank-aware FL methods do not personalize.
Critical strengths
- The paper has a clear taxonomy to organize the literature and what they are comparing against
- O4A is a simple and easy plug-and-play method for server-side aggregation
- Considerable improvements in performance over the baselines in the author's evaluation setup
Critical weaknesses
-
it is unclear if O4A is effective in real-world settings: 1a. the authors claim that O4A is effective across all four axes but did not evaluate on joint feature and label shift. In most real-world applications, all axes exist concurrently. 1b. the number of clients is typically orders of magnitude higher in real-world setups, possible implications: 1b (i). since k is clamp to the client with the lowest r, as the number of clients increase under heterogeneous rank, k will be fixed to a low value. 1b (ii). with many diverse clients, fewer directions get high agreement. 1b (iii). the consensus subspace would be different for each round under partial participation.
-
heterogeneous rank experiments is only compared with non-personalized FL methods. Are FlexLoRA or HetLoRA further finetuned? If not, it is an unfair comparison, comparing O4A to non-personalized FL works. Comparison to FedRoRA (Wang et al, 2026) is missing despite being mentioned in the Related Work.
-
k is a sensitive hyperparameter. Moreover, in practice, many existing works show that setting k to be fixed in every layer is suboptimal both in FL (Ramesh & Dess, 2026) and [1]) and LoRA rank sparsity works [2].
[1] Lee et al. "FedP²EFT: Federated Learning to Personalize PEFT for Multilingual LLMs.", AAAI 2026
[2] Zhang et al. "AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning", ICLR 2023
-
standard personalized FL baselines such as Ditto (Li et al, 2021), FedRep (Collins et al, 2021) are absent despite mentioned (they can be trivially applied to LoRAs at least for the homogeneous rank case)
-
the authors reported results for FedIT and LoRA-FAIR with no standard deviations from the LoRA-FAIR baseline, but is unclear if that number is comparable with their setup. this also contradicts 'All methods are re-implemented in a single pipeline' in their reproducibility statement.
Questions for authors
a) how does O4A perform under more realistic real-world settings with joint feature and label shift, orders of magnitude more clients, and partial participation?
b) why is local_only missing in most of the result tables? are the single global (non-personalized) FL baselines further finetune for personalization?