acceptodds
Withdrawn from ICLR 2027

FedCASA: Federated Consensus Alignment via Selective Aggregation in low-rank spaces

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

The rapid evolution of Large Language Models (LLMs) has driven widespread adoption of LoRA-based Federated Learning (FL), accompanied by increasingly diverse application demands. While some scenarios require a generalized model endowed with broad domain knowledge, others demand highly specialized models tailored to specific local tasks. However, existing federated LoRA frameworks struggle to simultaneously achieve both robust global generalization and effective local personalization. To address this gap, we propose FedCASA (Federated Consensus Alignment via Selective Aggregation). By exploiting the inherent rank independence and directional discrepancy of LoRA, our approach decouples the client-side LoRA updates into a global consensus component and a local personalized component. We evaluate both the derived global generalized model and the client-specific personalized models on multi-task instruction-tuning datasets across varying degrees of data heterogeneity. Extensive experiments demonstrate that FedCASA consistently outperforms existing baselines in both global and local performance, while simultaneously maintaining low communication costs. Our code is available at https://anonymous.4open.science/r/Phrolova-72.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Federated Learning, Large Language Model, Low-Rank Adaptation, Model Aggregation, Personalized Federated Learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_uZkgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how to obtain both a deployable global model and personalized local models through federated LoRA fine-tuning. FedCASA initializes LoRA using a truncated SVD of the pretrained weights. After the first training round, it uses the cosine similarity between each client’s columns of the B matrix and their corresponding initialization vectors to divide LoRA ranks into shared and private components through a greedy algorithm with a balancing constraint. The masks are then fixed: only the shared components are exchanged and aggregated, while the private components remain on each client. Using LLaMA-2 7B, the Dolly dataset, and eight clients, the authors report global-model results under three heterogeneity settings, local-model results under high heterogeneity, and analyses of communication cost, ablations, and convergence.

Critical strengths

Using directional similarity in the B matrix to select shared components at the rank level is a promising idea. The method uses the similarity between columns of B as its selection criterion and shares the corresponding A and B parameters, providing a concrete mechanism for separating shared and personalized components. Compared with aggregating all LoRA parameters uniformly, this approach allows different clients to retain different private components.

Critical weaknesses

1、FedCASA combines SVD initialization, similarity-based selection, and selective aggregation, but the paper should distinguish its substantive contributions more clearly from existing shared/private adapter and masked-aggregation methods. In particular, it should explain why remaining close to an initialization direction reliably indicates that a component contains knowledge suitable for sharing. The current ablations offer some support, but the independent benefit of this selection mechanism needs more direct validation under matched initialization and other conditions.

2、The experiments rely mainly on one model, one dataset, and eight clients. They do not assess larger client populations, different participation rates, or new-client scenarios. The current global evaluation primarily measures performance on the distributions of participating clients, so it does not adequately establish how well the model adapts to clients that did not participate in training or to new distributions. The paper also does not report variation across repeated runs, making it difficult to judge the stability of smaller performance differences.

Questions for authors

1、In Figure 2, the similarity matrix S assembled on the client side appears to have a different color arrangement from S used for greedy selection on the server side. Are these the same matrix? If so, the figure should use a consistent representation. If the matrix is reordered or otherwise processed between these steps, please explain how.

2、The global aggregation section of Figure 2 shows uploads from only three clients, whereas other parts of the figure refer to clients 1 through K. Are the remaining clients simply omitted for clarity? Please specify how many clients the figure depicts and explain, under partial client participation, how the number of contributors to each rank and the aggregation denominator are determined.

3、Could you add results with more clients, different client participation rates, and clients that did not participate in training to assess the method’s scalability and ability to generalize to new clients?

Reviewer_ZQmuOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes FedCASA for federated LoRA fine-tuning. After the first local training round, it uses cosine similarity between columns of the local and initial global matrices BB to select shared rank components. These components are aggregated, while the remaining components stay local. The method aims to produce both a global model and personalized local models with less communication. Experiments report improvements in both global and local performance.

Critical strengths
  1. The paper is well-structured, well-written, and easy to follow.
  2. The results are positive and show that FedCASA has the highest global score in all three tested data settings and the highest average personalized score.
  3. The paper includes useful comparisons with random rank selection, dynamic masks, and an analysis of the aggregation ratio.
Critical weaknesses
  1. The problem this paper proposes is valid, but not novel. There have been multiple FL and FedLoRA works that focus on both personalization and generalization. That being said, the contribution is incremental. For example, FedDPA trains a complete global adapter and local personalized adapters, and FedSA-LoRA separates shared and local knowledge through selective aggregation. Though they have been cited, the main new part of FedCASA is merely the selection of shared rank components within one adapter.
  2. Section 3 claims that LoRA rank components are strictly independent during training. However, their gradients depend on the same loss, which depends on all components. This is very important, as it affects the hold of the proposed framework.
  3. There is a serious problem in the convergence proof. Theorem 2 requires ceff>0c_{\mathrm{eff}}>0. However, Eq. (41) contradicts this condition and therefore would invalidate this convergence result.
  4. All experiments use LLaMA-2 7B, 8 clients, rank 8, and no random seeds. More ablation studies are needed to verify the scalability and robustness of the proposed framework.
  5. The global results and personalized results are obtained based on different settings. It does not clearly show that the same training run improves both global and personalized performance.
Questions for authors
  1. Can the authors justify W2 in theory?
  2. Can the authors show both global and personalized scores from the same training run, with the same training schedule for the compared baselines?
  3. Can the authors provide an additional ablation study regarding W4?
Reviewer_iSGuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

FedCASA aims to learn both global and personalized models through selective aggregation of rank-one LoRA components. Following SVD initialization, it scores each component by its alignment with the initial global direction and constructs a fixed sharing mask under client-wise and rank-wise quotas. Selected components are aggregated, while others remain local. Experiments on Dolly with LLaMA-2 7B evaluate global performance, personalization, communication, and component ablations.

Critical strengths
  1. The method provides a simple mechanism for retaining client-specific components while maintaining a complete global adapter. Selecting shared parameters at the rank-one level offers a concrete global–local trade-off.

  2. Experiments evaluate both global and personalized performance under different task partitions. The reported results and ablations support the effectiveness of the complete method within the LLaMA-2/Dolly setting, with lower communication than full LoRA transmission.

Critical weaknesses
  1. Fed-PLoRA [1] already uses parallel rank-one modules, subset-based local adaptation, and rank-wise aggregation. FedCASA differs in its fixed similarity-based selection and retention of private components, but should more clearly explain and evaluate the benefit of these differences. A comparison under compatible training and communication budgets would be informative.

  2. In Figure 1, the descending trend follows from sorting the cosine similarities. Variation across components does not establish that high-similarity components are more transferable or that low-similarity components are more useful for personalization. The random-selection ablation offers some support for the rule, but does not establish these functional roles.

  3. The selection score in Section 4.1 measures alignment with initialization rather than agreement among the clients being aggregated. Clients can remain similarly aligned with initialization while making conflicting updates, or simply making small updates. Moreover, Figure 1 uses the aggregated global direction, whereas the algorithm uses initialization; the connection between these references needs justification.

  4. In Section 3 (Equations 1–2), additive rank-one outputs do not imply strict rank independence: their gradients depend on the common loss derivative and are therefore coupled. SVD initialization does not by itself establish independence throughout training. The argument should be corrected, and its implications for treating components as separate global or local knowledge units clarified.

  5. The O(T^(-1/2)) convergence order in Section 4.2 and Appendix A.5 matches the nonconvex FedAvg rate cited by the authors. Under additional alignment and effective-descent conditions, it gives a conditional guarantee, but does not demonstrate improved constants or justify initialization similarity over alternative selection criteria. The scope of the theoretical contribution should be stated more precisely.

  6. Axis labels, legends, and other text within several figures are too small to read comfortably at normal viewing size. Please increase their font size.

[1] HETEROGENEOUS FEDERATED FINE-TUNING WITH PARALLEL ONE-RANK ADAPTATION, ICLR 2026.

Questions for authors

See the weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.