- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies how to obtain both a deployable global model and personalized local models through federated LoRA fine-tuning. FedCASA initializes LoRA using a truncated SVD of the pretrained weights. After the first training round, it uses the cosine similarity between each client’s columns of the B matrix and their corresponding initialization vectors to divide LoRA ranks into shared and private components through a greedy algorithm with a balancing constraint. The masks are then fixed: only the shared components are exchanged and aggregated, while the private components remain on each client. Using LLaMA-2 7B, the Dolly dataset, and eight clients, the authors report global-model results under three heterogeneity settings, local-model results under high heterogeneity, and analyses of communication cost, ablations, and convergence.
Critical strengths
Using directional similarity in the B matrix to select shared components at the rank level is a promising idea. The method uses the similarity between columns of B as its selection criterion and shares the corresponding A and B parameters, providing a concrete mechanism for separating shared and personalized components. Compared with aggregating all LoRA parameters uniformly, this approach allows different clients to retain different private components.
Critical weaknesses
1、FedCASA combines SVD initialization, similarity-based selection, and selective aggregation, but the paper should distinguish its substantive contributions more clearly from existing shared/private adapter and masked-aggregation methods. In particular, it should explain why remaining close to an initialization direction reliably indicates that a component contains knowledge suitable for sharing. The current ablations offer some support, but the independent benefit of this selection mechanism needs more direct validation under matched initialization and other conditions.
2、The experiments rely mainly on one model, one dataset, and eight clients. They do not assess larger client populations, different participation rates, or new-client scenarios. The current global evaluation primarily measures performance on the distributions of participating clients, so it does not adequately establish how well the model adapts to clients that did not participate in training or to new distributions. The paper also does not report variation across repeated runs, making it difficult to judge the stability of smaller performance differences.
Questions for authors
1、In Figure 2, the similarity matrix S assembled on the client side appears to have a different color arrangement from S used for greedy selection on the server side. Are these the same matrix? If so, the figure should use a consistent representation. If the matrix is reordered or otherwise processed between these steps, please explain how.
2、The global aggregation section of Figure 2 shows uploads from only three clients, whereas other parts of the figure refer to clients 1 through K. Are the remaining clients simply omitted for clarity? Please specify how many clients the figure depicts and explain, under partial client participation, how the number of contributors to each rank and the aggregation denominator are determined.
3、Could you add results with more clients, different client participation rates, and clients that did not participate in training to assess the method’s scalability and ability to generalize to new clients?