acceptodds
Under review as a conference paper at ICLR 2027

BaryLoRA: State-Conditioned Product Barycenters for Federated Low-Rank Adaptation

Abstract

Federated LoRA provides a communication-efficient way to adapt pretrained language models, but a shared adapter still requires the server to decide both what to aggregate and how to assign client contributions. Averaging low-rank factors can fail to aggregate the effective updates they encode: equivalent factorizations are non-unique, and averaging the factors separately introduces cross-client products. Meanwhile, sample-size weighting assigns each client the same relative mass at every adapted layer and therefore does not directly specify client-layer masses for a shared effective-update consensus. We introduce BaryLoRA, a state-conditioned product-space consensus framework that addresses these two mismatches jointly. BaryLoRA treats normalized effective updates as the consensus object and uses persistent optimization state to tilt a sample-size prior into client-layer contribution masses. The server then forms a weighted mean of the effective updates and returns its best rank-constrained approximation through a factorized computation that preserves the communication efficiency of low-rank adaptation. We distinguish the empirical state-to-mass design from the conditional properties of the resulting consensus. Under fixed positive state transforms and contribution masses, we establish uniqueness of the mass solution, invariance to equivalent LoRA factorizations, and optimality of the factorized rank-constrained realization. Experiments on language understanding, code generation, and mathematical reasoning tasks evaluate BaryLoRA against federated LoRA baselines.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Federated learning, Parameter-Efficient Fine-Tuning, Low-rank adaptation, Large language model

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_qKRNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes BaryLoRA, which studies how multiple clients can jointly fine-tune a large model while transmitting only small LoRA adapters and more effectively combining their training results. BaryLoRA aggregates the effective update matrices. It uses layer-wise activity summaries derived from optimizer states retained across communication rounds on each client to adjust the weights based on sample counts. Clients perform adaptive updates in the product space and maintain a low-rank representation through rank-constrained projection. The server forms a weighted mean of the effective updates using the adjusted weights, then applies a QR-core decomposition to compute its optimal low-rank approximation and produce a shared adapter. Communication consists mainly of low-rank factors and one state summary per layer; the full optimizer states remain on the clients.

Critical strengths

The method aggregates effective updates directly, avoiding the cross-client terms that arise when the two LoRA factors are averaged separately. Propositions 1 and 2 and Appendix A clearly explain the rank-constrained solution, its non-uniqueness when singular values are tied, and the equivalence between QR-core decomposition and dense projection. The numerical audit in Table 5 further supports the aggregation invariant under fixed weights. Together, these results make the method’s mathematical meaning and implementation objective relatively clear.

Critical weaknesses

1、BaryLoRA adjusts layer-wise aggregation weights using summaries of the second moment of each client’s historical gradients. Greater gradient activity, however, does not necessarily mean that an update is more valuable; it may also reflect data noise, task difficulty, or client drift. The paper explicitly treats this mapping as an empirical design choice. Its theoretical results establish the uniqueness of the weights given the state-based adjustment factors and the properties of aggregation under fixed weights, but they do not explain why the mapping should improve task performance.

2、BaryLoRA introduces several hyperparameters, including the strength of state-based weighting, the range used to clip the state summaries, and the local-prior coefficient. These directly affect how client contributions are weighted and how local optimization proceeds. The paper does not adequately explain how these values were chosen or whether the gains persist across a broad parameter range. In particular, removing the local prior causes substantial performance drops on some tasks, making the method’s dependence on prior strength worth examining. Without this evidence, it is difficult to judge whether the method requires careful tuning for each task or whether its gains will transfer reliably to new data distributions and federated settings.

Questions for authors

1、Figure 1 is described as an overview of BaryLoRA, but it mainly illustrates the limitations of factor averaging, the product-space representation, and weighted barycentric aggregation. It does not adequately show the client-side optimization procedure that is distinctive to this work. In particular, the figure omits the recovery of the update direction in product space, the first- and second-moment states retained across communication rounds, the rank-constrained projection after local updates, and how the state summaries are converted into layer-wise aggregation weights.

2、Although BaryLoRA keeps communication costs low by transmitting low-rank factors and a small number of state summaries, clients must still maintain a dense effective update matrix and first- and second-moment optimizer states, then perform a low-rank projection after local updates. These operations increase local memory and computation costs, potentially limiting the method’s applicability on resource-constrained devices.

Reviewer_yRDtOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies federated LoRA, mainly motivated by two problems: (1) factor-wise aggregation yields aggregation noise compared to the ideal aggregation of BA, (2) each adapted layer may need to have different weights. The first problem is well motivated and standard in prior work, authors address it via SVD reconstruction of the full rank updates (section 3.4). The second problem is motivated in a vague way, authors seem to propose an adam/momentum-like update rule (line 212-230). Experimental results show improvement than other baselines.

Critical strengths

Considering each client's contribution at each adapted layer is an interesting idea, while authors do not motivate this problem clearly in the paper. The experiments look good, BaryLoRA outperforms other federated LoRA methods. Section 2 (related work) is detailed and clear.

Critical weaknesses
  1. Presentation issues. "consensus object" is never formally defined, it is not a standard concept, authors use it multiple times without clear explanation of what it is. Some other concepts are also introduced without clear explanation, e.g., optimization state (in terms of what?), client-layer masses (do you mean weights?). Changing to standard language/concepts can help the community to understand the draft better. Overall, I admire authors' effort but Section 1's presentation is poor and very abstract for human readers. It is difficult to get any useful message about the algorithm details from line 66-80.
  2. Figure 1 is not clear, only Figure (C) explains the proposed algorithm, while Figure (A) and (B) are more like a summary of prior work. I don't understand how you present the difference between naive factor averaging and fedrot-lora. In Figure 1 (A), what's the difference between the vectors of client 1,2,3 (line 166) and the vectors of R1, R2, R3 (line 173)? For Figure 1 (B), I can somehow get your point that you want to aggregate (BA) instead of B and A, but your current presentation is hard for me to follow.
  3. Line 189 seems to be wrong.
  4. The presentation of Section 3 is poor. Authors describe the algorithm without any explanation to motivate why this makes sense.
Questions for authors
  1. How do you decide sls_l, is it something to optimize or a fixed parameter by the system?
  2. In section 3.4, you decompose μ\mu using SVD and only select the top rr components to build equation 14. I think that actually makes your algorithm very close to FlexLoRA and other SVD-based algorithms. Can you explain explicitly what is the difference between FlexLoRA and your algorithm? Reconstructing factors by cutting a full rank weights (μ\mu) to top-r is always a doubtful operation as you lose information. Can you explain how you algorithm avoid such information loss and also can you measure the information loss empirically (if possible)?
Reviewer_ZvtQOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes a layer-wise LoRA aggregation mechanism for Federated learning system.

Critical strengths

The paper consider Federated LoRA, one of the significant direction in recent literature to improve the learning efficiency.

Critical weaknesses
  • The paper presentations and writings require significant improvement. For instance, most of its components are proposed without explaining why they are needed. The notations are inconsistent, while many notations are being introduced without carefully used. Many notations are partly redundant.
  • As I learned from the method section, most components of BaryLoRA are established (e.g., Adam optimizer). Many components are presented with out explaining of why they are needed and novel. All of those drawbacks limit the paper from showing its novelty.
  • The research question of the paper (i.e., what to aggregate is already considered in various works, e.g., FlexLoRA, FLoRA, and FedEx-LoRA. Making the contributions and novelty very limited.
  • The results seem to be inconsistent among figures and tables. For instance, Fig. 2 and Table 1 have similar parameters but different results.
  • The second-moment summary is introduced without explanation, and the paper disclaims any interpretation of it as variance, precision, or reliability.
  • The literature reviews should be significantly improved, especially the subsection 2.2. Many references seem not to well support the layer-dependence client contributions. Meanwhile, some other references seem to be too old.
Questions for authors

I would like to ask the following questions to the author(s):

  • As in the Figure 1, the paper contributions seems incremental from FedRot-LoRA?
  • Why should a client with larger gradient magnitudes at a layer receive more weight?
  • Which configuration produced Fig. 2a/2c, and how does it reconcile with Table 1? Why does Table 3 report a different BaryLoRA value than Tables 1 and 8?
  • What is the usage of Q in Eq. (2)? Is it a simple copying the writing from FedRot-LoRA?
  • Could the authors clarify the definition of the concencus object? and justify what is it impact in the current paper's contribution?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.