acceptodds
Withdrawn from ICLR 2027

FedAS-LoRA: Adaptive Factor Sharing for Federated Low-Rank Adaptation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Low-rank adaptation (LoRA) represents large language model (LLM) updates with two compact matrix factors, i.e., and , providing an efficient way to fine-tune large models in federated learning paradigm. Inspired by the asymmetric roles of the LoRA factors, we study whether should be shared across clients while remains client-specific (Share-A/Local-B), or whether should instead be shared while remains client-specific (Share-B/Local-A). With a least-squares surrogate, we reveal that Share-A/Local-B requires the client-specific LoRA update matrices to use a common rank- input-side space, whereas Share-B/Local-A requires a common rank- output-side space. The two strategies therefore incur different projection residuals, indicating that the preferred strategy is the one with the smaller aggregate residual across clients. With this insight, we propose Federated Adaptive Factor Sharing Low-Rank Adaptation (FedAS-LoRA), which selects the sharing side before training to enhance fine-tuning performance. To enable adaptive factor selection before training, we design a Rank-Aware Shared-Subspace Sufficiency (RSS) metric, which effectively assesses whether a shared rank- input subspace is sufficient for the local data distributions using representations extracted from a frozen LLM backbone. Experiments across different tasks, data distributions, LoRA ranks, and participation settings confirm the effectiveness of RSS and the superior performance of FedAS-LoRA.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Low-Rank Adaptation, Federated Learning, Adaptive Factor Sharing

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_CKdPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In federated LoRA training, the authors reveal that Share-A/Local-B requires the client-specific LoRA update matrices to use a common rank-rinput-side space, whereas Share-B/Local-A requires a common rank-r output-side space. The two strategies therefore incur different projection residuals, indicating that the preferred strategy is the one with the smaller aggregate residual across clients. With this insight, they propose Federated Adaptive Factor Sharing Low-Rank Adaptation (FedAS-LoRA), which selects the sharing side before training to enhance fine-tuning performance. To enable adaptive factor selection before training, they design a Rank-Aware Shared-Subspace Sufficiency (RSS) metric, which effectively assesses whether a shared rank-rinput subspace is sufficient for the local data distributions using representations extracted from a frozen LLM backbone.

Critical strengths
  1. The paper is well written and easy to follow.
  2. The motivation of this work is clear.
  3. The finding about the difference between Share-A/Local-B and Share-B/Local-A is interesting.
Critical weaknesses
  1. There are some flaws in the method design. The current method is that, for an LLM, a fixed sharing policy is chosen without considering parameters related to federation, such as the number of clients and the data the client provides. It seems not reasonable.
  2. The result does not correspond to the description. For example, in Figure 1, the results of Share-B/Local-A are always better than Share-A/Local-B; why do the authors state, "This observation does not imply that Share-B/Local-A is always preferable." (Line 77). Are there any results where Share-A/Local-B is better than Share-B/Local-A?
  3. In Table 1, the authors' method, FedAS-LoRA, is superior to the previous baseline FedSA-LoRA. However, the authors' method only selects share A or share B. When the method selects share A, isn't it the same as FedSA-LoRA? Will the performance still be better than this baseline in this way? Or is it that the methods chosen by the authors in Table 1 are all share B?
Questions for authors
  1. How sensitive is the RSS choice to the number of clients, the non-IID settings, and the LoRA rank?
  2. Would choosing the sharing side separately for each layer or module work better than one global choice?
  3. How to demonstrate that RSS chooses the right sharing strategy?
Reviewer_rvQjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This work studies which of the two LoRA factors, A or B, should be shared across clients during federated fine-tuning. Its theoretical analysis shows that sharing A imposes a common input subspace, while sharing B imposes a common output subspace, making their effectiveness depend on client data. The method uses a Rank-Aware Shared-Subspace Sufficiency (RSS) metric computed from frozen LLM representations to select the sharing strategy before training. Experiments across tasks, data distributions, LoRA ranks, and client participation settings demonstrate the effectiveness of this adaptive approach.

Critical strengths
  1. This work provides both empirical and theoretical results.
  2. This work proposes a new adaptive sharing module strategy.
Critical weaknesses
  1. The motivation for factor sharing is unclear. Equation 1 alone does not sufficiently justify its necessity. Under FedAvg, if BiB_i and AiA_i converge to Bˉ\bar{B} and Aˉ\bar{A}, respectively, the two sides become equal. The paper should clarify when this discrepancy remains significant and why it warrants factor sharing.

  2. The theoretical analysis appears to rely on strong assumptions. Optimization is performed over the low-rank factors, whereas the convergence analysis is formulated in terms of the full adapter matrix. The connection between these two formulations needs stronger justification, including a discussion of the assumptions required for the analysis to hold.

  3. The theory does not clearly capture the adaptive sharing mechanism. The derivation appears equally applicable to FedSA, leaving it unclear what theoretical insight specifically supports the proposed adaptive strategy.

  4. The experimental evaluation is limited in scope and task complexity. The experiments primarily focus on GLUE, which provides insufficient evidence of the method’s effectiveness on more challenging tasks. Evaluation on more complex benchmarks would strengthen the empirical claims.

Questions for authors

See the weakness

Reviewer_8HWjOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies whether federated LoRA should share the input-side factor A or the output-side factor B. A least-squares analysis shows that Share-A/Local-B and Share-B/Local-A induce input- and output-side projection residuals, respectively. FedAS-LoRA then uses a training-free Rank-Aware Shared-Subspace Sufficiency (RSS) statistic, computed from frozen-backbone representations, to choose one policy before training. Experiments cover RoBERTa-large on GLUE and LLaMA3-8B on GSM8K under several ranks, data partitions, participation schemes, and LoRA variants.

Critical strengths
  1. The paper identifies a concrete and consequential design choice and gives it a clean projection interpretation. Eliminating the client-local factors in the least-squares surrogate makes explicit that Share-A/Local-B and Share-B/Local-A constrain different sides of the update. This explains why the preferred policy can vary with the client geometry and rank rather than being a universal convention. The analysis is simple, transparent, and directly motivates an adaptive method.

  2. The empirical study is unusually broad for federated PEFT. The paper evaluates several adaptation parameterizations, multiple tasks, ranks, partition types, participation distributions, and client counts, and reports means and standard deviations over three runs. The structured input-skew experiments and the explicit comparison of RSS decisions with both fixed policies are particularly relevant to the paper's hypothesis. Communicating one LoRA factor per round is also potentially useful if all other synchronized state and the one-time selector cost are accounted for.

Critical weaknesses
  1. The theory does not justify the implemented selector. Theorem 1 requires comparing residuals EAE_A and EBE_B​, but RSS measures only input-side sufficiency. Theorem 3 bounds EBE_B​ through an unobserved output-subspace dispersion and unknown constants, none of which appears in the decision rule. It also assumes that each desired update uses the leading eigenspace of SiS_i​ in the same coordinates. In practice, RSS uses one final CLS/last-token representation, while LoRA is applied to query and value projections across layers. The paper does not connect these spaces or justify one model-wide decision.

  2. Main tables omit fixed Share-B. Using Table 7's corresponding r=8, Dirichlet-0.5 values give about 90.54% for fixed Share-B versus 90.77% for FedAS-LoRA: a 0.24-point gain, not the emphasized 0.93 points over Share-A, with no paired uncertainty analysis.

  3. Algorithm 2 also appears to require sample-level representations at the server for random reassignment, but its communication, runtime, memory, and privacy costs are unreported. Table 12 lists about 1.84M trainable parameters but only 0.786M communicated for LoRA; the likely classification-head contribution is unexplained, so the claimed 50% end-to-end communication reduction cannot be verified.

Questions for authors
  1. Is one RSS decision used for every layer and projection? Please justify using final-token representations for intermediate query/value updates and report layer-wise alternatives or correlations with empirical EA−EBE_A−E_B​ and fixed-policy losses.
  2. Please include fixed Share-B in all main tables, report regret to the better fixed policy with paired confidence intervals, and clarify whether both policies received equal, independent hyperparameter tuning.
  3. What exactly is transmitted to compute RSS? Please report calibration replicates, ϵ\epsilon, bytes, runtime, memory, and privacy protection. Also clarify classification-head synchronization and include every synchronized tensor in communication costs.
  4. How can Theorem 2's per-round gradient-coverage and factor-alignment assumptions be guaranteed under the evaluated random sampling? If they cannot, please narrow the "arbitrary participation" claim.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.