acceptodds
Withdrawn from ICLR 2027

Same Updates, Different Aggregates: Rethinking Federated LoRA through Shared Subspaces

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Federated LoRA communicates low-rank factors to adapt models across decentralized clients. Equivalent factorizations preserve each local model, yet can produce different server aggregates under factor averaging. We formulate *gauge invariance* as an aggregation principle: equivalent representations of the same client updates should yield the same server update. Controlled audits of ordinary training uploads show that product-preserving refactorization can change predictions after factor averaging. Guided by this principle, we introduce GLoRA, a **G**auge-invariant aggregation framework for federated **LoRA** that separates client subspace geometry from update coordinates. Invariant client projectors define a shared reference space, giving client updates a common coordinate frame for aggregation. The resulting aggregate is gauge-invariant at finite server rank and recovers the exact update average under sufficient subspace coverage. A common low-rank server state supplies adapters for heterogeneous client ranks. Extensive experiments on GLUE and SuperNI cover homogeneous-rank, heterogeneous-rank, and task-heterogeneous settings, including sparse participation and unseen-task generalization. Compared with the strongest baselines in each setting, GLoRA improves homogeneous-rank GLUE averages by 0.58–0.93 accuracy points and unseen-task ROUGE-L by an average of 1.52 points across three SuperNI rank distributions, while reducing server aggregation time in the tested low-rank regimes.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Federated learning, low-rank adaptation, large language models

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_uaTfOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes GLoRA, a gauge-aware server representation for federated LoRA. The key idea is to use a invertible Q matrix to transform client B_i and A_i factors before aggregation. It creates a shared reference space and transforms LoRA factors towards it. GLoRA supports different client ranks. The paper does not provide rigorous theoretical support while the experimental results look promising, GLoRA outperforms other methods in different datasets across different federated settings.

Critical strengths
  1. The aggregation mismatch is an important topic for federated LoRA, the paper provides a good motivation.

  2. The experimental results are promising. GLoRA outperforms several fed LoRA baselines. Authors also include actual communication/delay cost results.

Critical weaknesses

The presentation of this paper is very poor and difficult for human readers to follow and understand.

  1. I don't see why authors explain their experimental setup in Section 2.2 (line 126) even before introducing the proposed algorithm. For Figure 1(b), how you measure the prediction change, is it the change of average accuracy? The current paper is very vague and unclear on that. For line 134, I don't see how SVD relates to the statement you want to make in this section. For line 145, "coordinate departure" is also an unclear concept without any definition. Authors should provide formal definitions of these metrics before showing them.

  2. For section 3, authors place all equations without any clear explanation. For Eq. 5, I think U_ref and Z_g are related to B, A, while the paper just simple states them as some "coordinates." Line 196: "The projector P=UiUi⊤P=U_iU_i^{\top} is unchanged by Bi→BiQiB_i\rightarrow B_i Q_i", I assume Q is related to U and Z, while the paper never explicitly explains their relationship. Line 198, TopEig_R is not a standard operator, which should be formally defined before use. Appendix (page 14), citing corresponding equations can help readers to understand the algorithm (please consider doing so). UitU_i^t seems to be the transformed version of BitB_i^t, then using B^ti\hat{B}_t^i instead of UtiU_t^i may improve the readability.

The major concern of this paper is the poor presentation from section 2.2 to 3.3.

Questions for authors
  1. Why do you compare GLoRA with FedSA-LoRA? FedSA-LoRA is a personalized fed LoRA, with no global model, how do you make the comparison fair?
  2. In Figure 3, seems GLoRA convergence is slower than FedEx-LoRA, can authors provide explanation on that?
Reviewer_LkusOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper observes that FedIT-style factor averaging is not invariant to equivalent refactorizations (BQ, Q⁻¹A) of client LoRA updates. It formalizes "gauge invariance" as a requirement for aggregation. It then proposes GLoRA, which works in four steps: 1) Take the top-R eigenspace of the weighted sum of client column-space projectors. 2) Project the updates onto it. 3) Average the coordinates, giving ΔW_g = P_ref M (Eq. 10). 4) Read out a rank-r_i adapter for each client, either by spectral order or by a history-based rule (CAR).

Critical strengths

Writing is clear and paper is easy to follow.

Critical weaknesses

The invariance thesis does not explain the gains, and the paper's own results argue against it. FlexLoRA (truncated SVD of M) is already gauge-invariant at every finite rank, because it depends only on M. The paper concedes this kind of invariance for update averaging and stacking. Therefore, the main heterogeneous-rank gains over FlexLoRA (Tables 2-3) cannot come from invariance. What does differ is how the rank-R subspace is chosen. Here GLoRA reconstructs M worse than SVD: 4.40% vs 1.19% error in Table 12, and 0.126 vs 0.020 in Table 16. Non-invariant HetLoRA beats invariant FlexLoRA in every Table 2 row, and non-invariant FedIT ties exact FedEx-LoRA in Table 1 (88.42 vs 88.43). The server budget used in the main tables is also never stated. The ρ = 1 values in Fig. 5 match Table 1 Dir(0.5) MNLI (0.857/0.857 vs 85.71/85.73) and Table 3 normal (0.523/0.380 vs 52.26/37.97). At ρ = 1, Prop. 2 gives ΔW_g = M, so GLoRA with spectral readout equals FlexLoRA's readout. In that case the gains would come from CAR or from implementation differences, not from the shared subspace. As written, the paper's causal story is unsupported.

The evidence that gauge dependence harms real training is weak, and part of it is internally inconsistent. The prediction flips arise only after an experimenter-imposed SVD refactorization. Natural gauge motion is small: 1.30–2.35% coordinate departure and about 1.56% gauge-tangent energy. The one training intervention (Table 10, single seed) moves FedIT accuracy by 0.22 points. Table 9 also reports FedIT aggregate changes of 8e-4 to 3e-3 together with 7.7–11.5% prediction disagreement. In Table 8, changes about 200× larger give 1.9-15% disagreement. That is implausible and needs checking. The efficiency claim has a similar problem: it compares against a dense FlexLoRA implementation. The truncated SVD of M can be computed from the stacked factors at GLoRA's own complexity, and GLoRA is still about 20× slower than FedIT (Table 27).

Other issues

  • FedSA-LoRA at Dir(0.1) scores 59.48 average, with QNLI 52.37, close to chance (Table 1). The evaluation protocol for this personalized method should be stated.
  • The text quotes heterogeneous gains against FlexLoRA (2.55/1.77/2.85). Against the strongest baseline, HetLoRA, they are 0.78/0.48/0.38. The abstract omits this setting from its "strongest baselines" numbers.
  • Several supporting analyses are single-seed.
  • Homogeneous GLUE uses only 3 clients with full participation.
  • The eigengap at R is never reported, although Table 11 (r_∪ = r_Σ) suggests the spectrum of K may be clustered.
  • LoRA-RITE already formalizes (BQ, Q⁻¹A) transformation invariance for LoRA optimizers. It is worth citing as precedent.
  • The server needs individual U_i, so compatibility with secure aggregation is not discussed.
Questions for authors
  • Please add FLoRA (stacking: exact, invariant, heterogeneous ranks) to Tables 2-3 and time FlexLoRA through a factored SVD
  • Please verify Table 9. How can update changes of about 1e-3 flip about 10% of the 872 predictions when changes of about 0.5 in Table 8 flip 2-15%?
Reviewer_uwexOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces GLoRA, a gauge-invariant aggregation for federated LoRA that constructs a shared subspace from client update geometry to avoid representation-dependent aggregation. GLoRA supports heterogeneous client ranks and demonstrates improved performance and server-side aggregation efficiency.

Critical strengths

S1. The part of proposing Gauge Invariance as the aggregation principle of Federated LoRA is clearly formulated and mathematically sound. However, update averaging itself also possesses Gauge Invariance, so it is more like a unified explanation and design principle for the known aggregation behavior.

S2. "Proposing a low-rank aggregation method based on client subspace geometric consensus" This part has a clear algorithm design and independent ablation support. I believe this is the more substantive technical contribution of this paper.

Critical weaknesses

W1. In LoRA training, the factors AA and BB are only optimized, without introducing gauge transformations QQ. While the paper demonstrates sensitivity to artificially constructed equivalent factorizations, it remains unclear to what extent such gauge discrepancies arise naturally during federated optimization. Could the authors provide a direct empirical analysis that isolates the effects of naturally occurring gauge misalignment from genuine differences in client updates?

W2. The paper reports server-side speedups over FlexLoRA (up to 38.3\times), which constitute an important part of its practical contribution. However, the comparison appears to rely on a dense implementation of update averaging followed by SVD.

Since the averaged update can be expressed directly as a product of concatenated low-rank factors,

M=∑i=1npiBiAi=[p1B1,p2B2,⋯ ,pnBn][A1,A2,⋯ ,An]⊤M = \sum_{i=1}^{n} p_i B_i A_i = [ p_1 B_1, p_2 B_2, \cdots, p_n B_n ] [ A_1, A_2, \cdots, A_n ]^\top

an efficient low-rank SVD can be computed without explicitly materializing the full update matrix. Such an implementation could significantly reduce the computational cost of the FlexLoRA baseline.

So I concern that the reported speedups may partly reflect implementation differences rather than an inherent computational advantage of GLoRA.

Questions for authors

Q1. The author needs to explain why choosing the direction based on the geometric consensus of the client subspaces may be superior to minimizing the reconstruction error of the average update in federated optimization. This cannot be answered merely by referring to Gauge Invariance, as both methods can possess Gauge Invariance.

Reviewer_gfHxOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes GLoRA, which aggregates federated LoRA updates in a shared subspace. To do so, the server builds a shared subspace from the clients' subspaces, and then averages the updates within it.

Critical strengths

The experimental evaluation is comprehensive. Specifically, the paper considers different heterogeneity settings in the experiments.

Critical weaknesses
  • The paper presentation and writing require significant improvement. For example, the motivation does not explain the contribution. Furthermore, many notations are introduced without being used. The figure 1 do not bring much insights. comparing to recent literature (e.g., SSA LoRA).
  • The paper novelty seems to be limited. For instance, the main contribution lies in the projection-based subspace construction, while it is well considered in various works (including Federated Orthogonal Training). It seems the current paper contribution is incremental by just applying those projection-based training to LoRA than propose a new method. Furthermore, the authors do not discuss the difference and improvement over the recent works, e.g., FlexLoRA and FLoRA.
  • Given the similarity with the projection-based learning and orthogonal training, the literature reviews according those lines of works are limited.
  • The theoretical guarantees establish invariance, but do not explain the learning benefit of selecting directions by subspace overlap.
Questions for authors
  1. Could the authors clarify what they mean by “gauge invariance” in this context? In particular, when stating that the aggregation “should return the same update,” do the authors mean that standard aggregation of LoRA factors introduces an error, or a mismatch between the aggregated factors and the aggregated full update? A short example would help readers follow this point.
  2. Could the authors give a formal definition, or an intuitive explanation, of “representation freedom,” and explain how it relates to the proposed method?
  3. How is the “invariant client” identified among the participating clients? It would help if the authors could describe the selection criterion or procedure, and say whether it depends on client heterogeneity or participation rate.
  4. Could the authors elaborate on the distinction between “where an update lies” and “its coordinates within that space”? A more precise mathematical statement, for example in terms of the column space of the update versus its basis representation, would make this clearer.
  5. Could the authors explain the role of the invertible matrix Q?
  6. Could the authors define the term “factor pairs” explicitly?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.