- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper addresses federated LoRA fine-tuning when clients use different ranks (4 to 64). Building on the pad-to-maximum-rank, average and truncate framework, it proposes FedRRA with two parts:
CAP (Consensus-Aware Padding) fills a low-rank client's missing k-th dimension with a weighted average of the k-th components of clients that have it. A client's weight is the magnitude of its component times how much its direction agrees with the other clients, and only clients with above-average agreement contribute. ARC (Aggregation Residual Correction) has the server compute the difference between the average of the clients' BA products (the ideal update) and each client's truncated LoRA product, compress it with a rank-16 SVD, and send it to the client to add to its frozen weights.
On TinyLlama, Qwen2.5-3B and Llama2-7B fine-tuned on Dolly, Alpaca and Wizard, FedRRA scores higher than the baselines on MMLU and MT-Bench in every setting.
Critical strengths
- Heterogeneous client ranks are a practical problem in federated fine-tuning, and padding-based methods are simple and cheap.
Critical weaknesses
- The improvements are small and the evaluation discriminates poorly. Only three seeds are used and no significance tests are reported. Base-model scores before fine-tuning are not reported, so it is impossible to tell how much fine-tuning changes the metrics.
- The server cost is high. The introduction criticizes FlexLoRA for large-scale SVD with extra computation and decomposition error, yet FedRRA performs an SVD with truncation for every client's matrices in every round, which also incurs truncation error. In Table 16, FedRRA's server time per round on TinyLlama is 27.78 s against 0.51 s for HETLoRA and 17.89 s for FlexLoRA, and its communication (92.36M parameters) exceeds that of HETLoRA and FlexLoRA (69.84M). So the motivation conflicts with the method.
- The models used in the experiments are old. The setting is narrow. Training runs for only 4–5 rounds with 10 clients, and LoRA is applied only to q and v.
Questions for authors
- Can you use more advanced models?
- Can you add homogeneous baselines in which all clients use rank, and exact aggregation combined with FLoRA or FlexLoRA?
- Can you add seeds and run significance tests?
- The server performs SVDs for every client's matrices each round. How does server time grow for models beyond 7B and for hundreds of clients?