acceptodds
Under review as a conference paper at ICLR 2027

What Should Federated LoRA Share? FedSAIL via Input-aware Subspace Alignment

Abstract

Federated low-rank adaptation (LoRA) requires identifying an update structure that is genuinely shared across heterogeneous clients. Prior work reports strong similarity among trained LoRA projection matrices across clients; however, such agreement may be largely induced by common initialization and collapses toward random overlap under independent initialization. More crucially, relying solely on parameter similarity inherently ignores the influence of local input regime. To uncover a more robust shared structure, we introduce an input-aware action matrix that weights the adapter update by the second-moment statistics of local layer inputs. Empirically, while parameter similarity vanishes, the leading right singular directions of this action matrix remain strongly aligned across clients. This shared geometry preserves task-conditioned differences and naturally varies across network depths. Motivated by these findings, we propose Federated Subspace-Guided Action-Informed Learning (FedSAIL). Instead of averaging weights, FedSAIL estimates a shared action subspace to regularize local training while preserving client-specific coefficients. Across several benchmarks, our approach consistently improves predictive performance over competing federated LoRA methods while reducing communication cost significantly.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
federated learning, low-rank adaptation, parameter-efficient fine-tuning, subspace sharing, communication efficiency

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_WPTdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes an input-aware LoRA action matrix to identify shared subspaces between clients. FedSAIL guides local training by sharing spectral information while preserving personalized parameters. The experiments primarily validated accuracy and communication benefits under GLUE and cross-task settings.

Critical strengths
  1. Distinguishing between parameter similarities arising from initialization and shared structures formed during training is a meaningful endeavor, and the experimental results offer valuable insights.

  2. The design that decouples the shared subspace from local parameters is elegant, allowing for increased local adaptation capacity without increasing communication overhead.

Critical weaknesses
  1. The baseline comparison is insufficient. The main experiments consistently employ independent initialization, whereas FFA-LoRA deviates from its standard "shared-frozen-A" configuration. The absence of a comparison under the standard setting makes it difficult to assess the method's advantage over the baseline operating in its intended mode.

  2. The scope of validation is relatively narrow. The main experiment has only three clients per task, and the cross-task experiment has only eight clients, which is insufficient to support the effectiveness in general federated scenarios. The average accuracy gains are small, and there is a lack of validation on a larger scale and with the participation of some clients.

Questions for authors
  1. Does FedSAIL maintain its advantages under the settings of joint initialization and standard FFA-LoRA?

  2. Do performance and communication gains remain stable when the number of clients is increased and partial participation is employed?

2 comments
AuthorsOpenReview ↗

Response to Reviewer WPTd: W1 and Q1

Thank you for pointing this out. We agree that evaluating FFA-LoRA under its standard configuration, where clients share a common frozen initialization of AA, is essential for a fair comparison. Following your suggestion, we evaluated FFA-LoRA under common initialization across the GLUE benchmark and also included the corresponding results for FedSA-LoRA, as reported below.

\hline \text{Validation accuracy on GLUE benchmark (\\%), mean } \pm \text{ sample SD} \\\\ \hline \begin{array}{cllccccccc} \alpha & \text{Method} & \text{Init.} & \text{RTE} & \text{SST-2} & \text{QNLI} & \text{MNLI-m} & \text{MNLI-mm} & \text{QQP} & \text{Avg.} \\\\ \hline 0.50 & \text{FedSA-LoRA} & \text{Ind.} & 79.90_{\pm 8.61} & 95.95_{\pm 1.13} & 91.76_{\pm 0.61} & 88.56_{\pm 1.72} & 88.92_{\pm 1.76} & 89.53_{\pm 5.63} & 89.10_{\pm 1.60} \\\\ & \text{FedSA-LoRA} & \text{Common} & 80.87_{\pm 7.51} & 96.14_{\pm 1.36} & 92.20_{\pm 0.56} & 88.81_{\pm 1.70} & 88.88_{\pm 1.72} & 89.54_{\pm 5.68} & 89.40_{\pm 1.34} \\\\ & \text{FFA-LoRA} & \text{Ind.} & 70.04_{\pm 5.91} & 90.75_{\pm 1.78} & 79.46_{\pm 2.06} & 65.46_{\pm 9.63} & 66.65_{\pm 9.11} & 81.28_{\pm 11.53} & 75.61_{\pm 3.27} \\\\ & \text{FFA-LoRA} & \text{Common} & 70.88_{\pm 3.47} & 95.15_{\pm 1.03} & 84.86_{\pm 2.21} & 84.02_{\pm 3.63} & 84.83_{\pm 3.71} & 86.78_{\pm 6.78} & 84.42_{\pm 1.17} \\\\ \hline 0.75 & \text{FedSA-LoRA} & \text{Ind.} & 76.65_{\pm 3.86} & 95.41_{\pm 0.41} & 92.34_{\pm 0.29} & 88.22_{\pm 1.58} & 88.42_{\pm 1.75} & 88.70_{\pm 4.87} & 88.29_{\pm 0.39} \\\\ & \text{FedSA-LoRA} & \text{Common} & 77.14_{\pm 2.94} & 95.72_{\pm 0.59} & 92.89_{\pm 0.42} & 88.39_{\pm 1.58} & 88.68_{\pm 1.66} & 89.00_{\pm 4.66} & 88.63_{\pm 0.13} \\\\ & \text{FFA-LoRA} & \text{Ind.} & 61.13_{\pm 3.47} & 89.22_{\pm 2.81} & 81.77_{\pm 2.42} & 62.33_{\pm 10.84} & 63.60_{\pm 10.45} & 81.53_{\pm 8.88} & 73.26_{\pm 1.81} \\\\ & \text{FFA-LoRA} & \text{Common} & 64.02_{\pm 2.66} & 94.69_{\pm 0.24} & 86.54_{\pm 0.82} & 83.58_{\pm 2.72} & 84.45_{\pm 3.00} & 86.05_{\pm 5.41} & 83.22_{\pm 0.96} \\\\ \hline 1.00 & \text{FedSA-LoRA} & \text{Ind.} & 79.54_{\pm 4.35} & 95.83_{\pm 0.81} & 92.32_{\pm 0.66} & 88.10_{\pm 1.52} & 88.17_{\pm 1.60} & 88.18_{\pm 2.14} & 88.69_{\pm 1.31} \\\\ & \text{FedSA-LoRA} & \text{Common} & 79.78_{\pm 3.08} & 95.83_{\pm 1.03} & 92.71_{\pm 0.42} & 88.04_{\pm 1.48} & 88.17_{\pm 1.55} & 88.29_{\pm 2.15} & 88.81_{\pm 1.06} \\\\ & \text{FFA-LoRA} & \text{Ind.} & 76.29_{\pm 4.22} & 87.88_{\pm 0.98} & 80.08_{\pm 1.36} & 61.48_{\pm 11.53} & 62.48_{\pm 10.99} & 81.69_{\pm 4.80} & 74.98_{\pm 4.05} \\\\ & \text{FFA-LoRA} & \text{Common} & 76.41_{\pm 4.34} & 94.76_{\pm 0.98} & 87.39_{\pm 1.21} & 83.18_{\pm 2.92} & 83.84_{\pm 2.76} & 85.25_{\pm 2.62} & 85.14_{\pm 1.92} \\\\ \end{array} \\\\ \hline \end{array}$$ As shown in the table above, indeed, adopting common initialization improves FFA-LoRA's average accuracy by approximately $9\text{-}10$ percentage points across the GLUE benchmark, as a shared frozen $A$ provides a consistent projection basis for aggregating $B_k$. Motivated by this observation, we also examined the impact of initialization on FedSA-LoRA. Although the official implementation of FedSA-LoRA uses common initialization, our results indicate that its performance is largely insensitive to the initialization scheme, since $A_k$ is actively trained and synchronized in every round. Overall, even after updating the baselines to their common-initialization results, our FedSAIL approach still maintains its performance advantage while substantially reducing communication costs.
AuthorsOpenReview ↗

Response to Reviewer WPTd: W2 and Q2

Thank you for this important suggestion. To evaluate scalability and robustness under partial participation, we extended our RoBERTa-large experiments on GLUE (α=0.75\alpha=0.75) to federations of 10, 20, and 100 clients over 1,000 communication rounds, with a client sampling rate of 0.3 per round. Due to computational constraints during the rebuttal period, we first report validation accuracy from a single seed (seed 123) below, and will include full multi-seed means and standard deviations in the revised manuscript.

\begin{array}{lccccccccccccccc} \hline \text{Method} & & \text{QNLI} & & & \text{SST-2} & & & \text{MNLI-m} & & & \text{MNLI-mm} & & & \text{Avg.} & \\\\ \hline & 10 & 20 & 100 & 10 & 20 & 100 & 10 & 20 & 100 & 10 & 20 & 100 & 10 & 20 & 100 \\\\ & \text{clients} & \text{clients} & \text{clients} & \text{clients} & \text{clients} & \text{clients} & \text{clients} & \text{clients} & \text{clients} & \text{clients} & \text{clients} & \text{clients} & \text{clients} & \text{clients} & \text{clients} \\\\ \hline \text{Local-LoRA} & 89.18 & 87.59 & 88.03 & 92.78 & 93.23 & 93.00 & 86.45 & 84.35 & 84.35 & 86.58 & 84.97 & 84.80 & 88.75 & 87.54 & 87.55 \\\\ \text{FFA-LoRA} & 89.27 & 89.29 & 86.95 & 95.07 & 95.30 & 95.07 & 88.75 & 87.44 & 86.35 & 89.13 & 87.79 & 87.14 & 90.56 & 89.96 & 88.88 \\\\ \text{FedSA-LoRA} & 90.92 & 92.17 & 90.01 & 94.84 & 94.72 & 94.50 & 90.23 & 87.29 & 84.20 & 90.99 & 87.85 & 84.52 & 91.74 & 90.51 & 88.30 \\\\ \hline \text{FedSAIL} & 92.17 & 91.98 & 89.91 & 94.95 & 95.99 & 94.27 & 90.86 & 89.58 & 88.04 & 90.99 & 90.15 & 88.35 & 92.24 & 91.93 & 90.14 \\\\ \hline \end{array}$$ As shown in the results above, FedSAIL's performance advantage over the baselines becomes even more pronounced as the number of clients increases under partial participation, with its average accuracy margin over FedSA-LoRA widening from $+0.50\\%$ at 10 clients to $+1.42\\%$ at 20 clients and $+1.84\\%$ at 100 clients. With smaller local datasets and partial participation, directly overwriting local parameters with partial averages (as in FFA-LoRA and FedSA-LoRA) can destabilize local adaptation. In contrast, FedSAIL keeps both LoRA factors local and guides training via soft subspace regularization, allowing it to remain more stable against client drift. $$\small \begin{array}{lrrr} \hline \text{Method} & \text{10 clients} & \text{20 clients} & \text{100 clients} \\\\ \hline \text{FedSAIL (50 rounds warmup)} & 1.0447 & 2.0894 & 10.4472 \\\\ \text{FFA-LoRA} & 8.7891 & 17.5781 & 87.8906 \\\\ \text{FedSA-LoRA} & 8.7891 & 17.5781 & 87.8906 \\\\ \hline \text{Communication reduction} & 88.11\\% & 88.11\\% & 88.11\\% \\\\ \hline \end{array}$$ **Table 1:** Total bidirectional communication costs per run (GiB) over 1,000 rounds with a 0.3 sampling rate ($p=1024, r=8, s=1$, 4-byte entries). For communication efficiency, since each participating client's per-round payload is independent of the federation size, the total communication volume scales linearly with the number of sampled clients per round. As shown in Table 1, FedSAIL consistently reduces total communication cost by 88.11% across all client scales compared to both baselines.
Reviewer_RqGdOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors introduce FedSAIL, a federated learning method that finds a common subspace induced across clients by LoRA fine-tuned models. First, it argues via controlled diagnostics that the high cross-client similarity of trained A matrices factors is not induced by fine-tuning, but rather from a common initialization, verifying so using a Projector Dice experiment that shows how different LoRA initializations make the similarity fall to random baseline. Then, the authors introduce an input-aware matrix Gk=BkAkΣk1/2G_k = B_kA_k\Sigma_k^{1/2}, with Σk\Sigma_k being the uncentered second moment of layer inputs. The top right singular direction of GkG_k remain aligned across clients even with independent initializations, also showing task and depth-dependent agreement across clients. Based on these observations, FedSAIL is designed to let the clients upload, at the end of the first round after warm-up, the top s eigenpairs of Gk⊤GkG_k^\top G_k, computed in a more efficient way to save memory and computation. The server computes the trace-normalized Gram matrix of rank s of these subspaces and aggregates these Gram matrices using weighted averaging, and its top eigenspace is selected, resulting in V^\hat V. Finally, an orthogonal projector P^=V^V^⊤\hat P=\hat V\hat V^\top is computed and broadcasted back to all clients, which can start the second round using this quantity to regularize their local training objective by penalizing the energy, i.e. BkAkΣk1/2B_kA_k\Sigma_k^{1/2}, that falls outside of this projector’s subspace. The method continues with this 2 components (upload of the sub-space and regularization during fine-tuning) until the last communication round. Finally, layers which show low cross-clients agreement (typically the last layers since they are more specialized) can disable sharing. Experiments on personalized GLUE (RoBERTa-large, 3 clients per task, Dirichlet label skew) show a mean accuracy of 89.33% vs 88.69% for FedSA-LoRA. Further experiments cover shared/local rank sweeps, a shared-message ablation, and a DDDT layer-selection study reporting a 19.26% communication reduction at comparable accuracy.

Critical strengths
  1. A useful, well-controlled diagnostic of a widely cited motivation. The demonstration that AA-similarity collapses to the random level (0.0105≈8/7680.0105 \approx 8/768) under independent initialization, measured with a basis-invariant metric, is a clear and reusable negative result for the federated LoRA literature.

  2. A principled and communication-light design. The shared object is invariant to LoRA reparameterization, aggregation is done on PSD matrices rather than on sign/rotation-ambiguous eigenvectors, and the payload depends only on ss, not on the local rank rLr_L.

Critical weaknesses
  1. The central claim, that gains come from sharing input-aware geometry, is not isolated.

    • Several observations suggest the effect may be driven by the frozen backbone's input anisotropy and by local low-rank regularization rather than by collaboration. The leading direction carries ~97% of action energy, Σk\Sigma_k is uncentered (and so dominated by the hidden-state mean/outlier directions), and the penalty forces action into a rank-1 subspace. This can be even more true given transformer hidden states (especially RoBERTa’s), which tend to have a large mean μ\mu, making μμ⊤\mu\mu^\top quite large.

    • The ablation in Table 4 reinforces this concern rather than resolving it. Messages from AkA_k or Δk\Delta_k yield ~68%, about 20 points below purely local LoRA (88.5%). This shows that the G^k\hat G_k-based penalty is destructive when P^\hat P is mismatched, not that input-aware sharing is beneficial.

  2. The empirical evaluation is too narrow to support the claims of consistent superiority.

    • The setting is a single encoder model (RoBERTa-large) on GLUE classification, with 3 clients per federation and full participation, which is too simple of a setting for a federated learning manuscript.

    • There are no decoder-LLM or generative tasks, which is not only a more competitive setting, but also the standard setting for the top competitors in this literature.

    • The gains over local LoRA (+0.8 points on average) and FedSA-LoRA (+0.6) are within roughly one seed standard deviation, and no significance tests are reported.

    • All baselines are run under independent initialization, which is non-canonical for FedSA-LoRA and FFA-LoRA; the authors themselves admit that this is a stress test for FFA-LoRA. It is fine to show the facade of similarities under same and independent initialization, but competitors should also be evaluated according to their original configuration if the setting allows it, and this setting definitely allows it.

    • Hyperparameters (especially the learning rate since there appears to be a single one of 0.005) appear shared rather than tuned per method, and the motivation about the selection of the specific hyperparameters is missing.

    • Methods discussed in the paper (FedRot-LoRA, LoRA-FAIR, FLoRA, FedIT, i.e. FedAvg-LoRA) and proximal personalization (e.g., Ditto with LoRA) are not compared.

    • The abstract claims a significantly reduced communication overhead, but this is not backed by a payload comparison against the baselines.

Questions for authors
  1. What is the accuracy of FedSAIL with the communication step removed, i.e., each client regularized toward its own top-ss action subspace? And with P^\hat P set to the top eigenvector(s) of Σ^k\hat\Sigma_k or of the averaged Σ^\hat\Sigma? If these match FedSAIL, what is the role of federation?

  2. In the diagnostics, how much of the top-1 GkG_k agreement is explained by agreement of the top eigenvectors of Σk\Sigma_k across clients? Does the picture change with a centered covariance?

Suggestions for the authors:

  1. Add a decoder-LLM experiment (e.g., Llama-family instruction tuning or math reasoning) with more clients, optionally also with partial participation, and more related competitors (see Weaknesses).

  2. Discuss what information about client inputs is revealed by Σ\Sigma-weighted directions, and by the selector's per-token-regime snapshots, relative to sharing raw factors.

10 comments
AuthorsOpenReview ↗

Response to W1.1, Q1, Q2, and S2 (Part 1/3 of this response) — Reviewer RqGd (Part 1/8)

Thank you for highlighting the potential roles of backbone input anisotropy and local low-rank regularization in W1.1, and for proposing the targeted controls in Q1 and Q2 alongside the input information discussion in S2. We will address each point below.

First, regarding why FedSAIL guides regularization through GkG_k rather than input anisotropy alone (W1.1 and Q1), recall that Σk\Sigma_k describes only the energy distribution of frozen-backbone layer inputs, whereas Gk=ΔkΣk1/2G_k = \Delta_k \Sigma_k^{1/2} also incorporates the learned adapter's response Δk\Delta_k. Because a dominant input direction may already be well handled by the pretrained backbone model, downstream adaptation often requires corrections along a different direction. While a formal analysis for deep language models is intractable, we can construct a single-layer regression example under squared loss to illustrate why targeting GkG_k is preferable. Specifically, let YkY_k denote the required correction and suppose Yk=Δk⋆h+εkY_k = \Delta_k^\star h + \varepsilon_k, with finite second moments, Ek[εk∣h]=0\mathbb{E}_k[\varepsilon_k \mid h] = 0, and Σk=Ek[hh⊤]≻0\Sigma_k = \mathbb{E}_k[hh^\top] \succ 0. Writing Gk⋆=Δk⋆Σk1/2G_k^\star = \Delta_k^\star \Sigma_k^{1/2} and Lk(Δ)=12Ek[∣Yk−Δh∣22]L_k(\Delta) = \tfrac{1}{2}\mathbb{E}_k\bigl[\\|Y_k - \Delta h\\|_2^2\bigr], for any fixed rank-ss orthogonal projector PP and β>0\beta > 0, let Δk,P\Delta_{k,P} minimize

L_k\bigl(\Delta\bigr) + \frac{\beta}{2}\bigl\\|\Delta\Sigma_k^{1/2}\bigl(I - P\bigr)\bigr\\|_F^2.

Completing the square with respect to ΔΣk1/2\Delta\Sigma_k^{1/2} yields the excess prediction risk

= \frac{\beta^2}{2\bigl(1+\beta\bigr)^2}\bigl\\|G_k^\star\bigl(I - P\bigr)\bigr\\|_F^2.$$ Among all rank-$s$ projectors, this excess risk is minimized by the leading right singular projector of $G_k^\star$, and the same result holds under any LoRA rank sufficient to represent $\Delta_k^\star$. Guiding regularization with $G_k^\star$ therefore achieves strictly lower risk than guiding with the top eigenvectors of $\Sigma_k$ whenever it captures more true action energy. In practice, each client estimates its action subspace from finite samples with noise, so federated pooling can reduce subspace estimation error for a shared correction direction while leaving client-specific output coefficients unconstrained.
AuthorsOpenReview ↗

Response to W1.1, Q1, Q2, and S2 (Part 2/3 of this response) — Reviewer RqGd (Part 2/8)

Second, to empirically test these effects on prediction performance on the GLUE benchmark (W1.1 and Q1), we compare FedSAIL against three controls where the guidance projector P^\widehat{P} is instead constructed from each client's local input moment (Local Σ^k\widehat{\Sigma}_k), the averaged input moment (Averaged Σ^\widehat{\Sigma}), or each client's own action subspace without communication (Local-GkG_k). Due to limited computational resources during the rebuttal period, we first report the results on seed 123123 below:

\hline \text{Validation accuracy on GLUE benchmark (\\%)} \\\\ \hline \begin{array}{clccccccc} \alpha & \text{Method} & \text{RTE} & \text{SST-2} & \text{QNLI} & \text{MNLI-m} & \text{MNLI-mm} & \text{QQP} & \text{Avg.} \\\\ \hline 0.50 & \text{Local }\widehat{\Sigma}_k & 84.12 & 94.95 & 92.93 & 86.66 & 86.80 & 85.32 & 88.46 \\\\ & \text{Averaged }\widehat{\Sigma} & 84.12 & 95.30 & 92.73 & 86.81 & 86.85 & 85.66 & 88.58 \\\\ & \text{Local-}G_k & 83.39 & 95.41 & 92.86 & 87.35 & 87.26 & \mathbf{86.19} & 88.74 \\\\ & \text{FedSAIL} & \mathbf{85.20} & \mathbf{95.53} & \mathbf{93.37} & \mathbf{87.56} & \mathbf{87.55} & 85.92 & \mathbf{89.19} \\\\ \hline 0.75 & \text{Local }\widehat{\Sigma}_k & \mathbf{83.39} & 94.95 & 92.29 & 88.32 & 88.57 & 84.83 & 88.73 \\\\ & \text{Averaged }\widehat{\Sigma} & 83.03 & 95.07 & 92.26 & 88.27 & 88.52 & 84.42 & 88.59 \\\\ & \text{Local-}G_k & 81.59 & 95.07 & 92.88 & 89.03 & 89.28 & 85.64 & 88.91 \\\\ & \text{FedSAIL} & 83.03 & \mathbf{95.53} & \mathbf{93.26} & \mathbf{89.06} & \mathbf{89.37} & \mathbf{85.67} & \mathbf{89.32} \\\\ \hline 1.00 & \text{Local }\widehat{\Sigma}_k & 75.81 & 95.87 & \mathbf{93.08} & 87.28 & 87.36 & 85.96 & 87.56 \\\\ & \text{Averaged }\widehat{\Sigma} & 75.81 & 95.87 & 92.18 & 87.20 & 87.19 & 85.54 & 87.30 \\\\ & \text{Local-}G_k & 76.53 & 95.99 & 92.84 & 88.39 & 88.22 & 86.30 & 88.05 \\\\ & \text{FedSAIL} & \mathbf{78.34} & \mathbf{96.44} & 92.84 & \mathbf{88.65} & \mathbf{88.44} & \mathbf{86.63} & \mathbf{88.56} \\\\ \end{array} \\\\ \hline \end{array}$$ As shown in the table above, first, although the margins are modest, full FedSAIL outperforms Local $\widehat{\Sigma}_k$ and Averaged $\widehat{\Sigma}$ by $0.77$ and $0.87$ percentage points, indicating that regularizing solely along the input $\widehat{\Sigma}_k$ directions is less effective than action-based guidance. Second, we agree that part of the improvement in prediction accuracy comes from regularizing $G_k$ locally. In our response to Reviewer pmgh ([Table R1](https://openreview.net/forum?id=AwjS0mv7WU&noteId=ory8ElNoeB)), we provide detailed three-seed results showing that Local-$G_k$ already gains $1.55$ points on average over unregularized Local-LoRA. In comparison, full FedSAIL improves over Local-$G_k$ by only $0.25$ points across the three-seed average, with larger benefits when per-client data are limited and local subspace estimates are noisier.
AuthorsOpenReview ↗

Response to W1.1, Q1, Q2, and S2 (Part 3/3 of this response) — Reviewer RqGd (Part 3/8)

Third, regarding Q2 and the effect of the hidden-state mean μk\mu_k raised in W1.1, we agree that a large shared input mean contributes strongly to uncentered alignment. To examine how much of the top-11 GkG_k agreement comes from the leading eigenvector of Σk\Sigma_k and how centering changes the results, we re-evaluated the trained RoBERTa-base/RTE and GPT-2 checkpoints from our common-initialization diagnostics by replacing the uncentered second moment Σk=Ek[hh⊤]\Sigma_k = \mathbb{E}_k[hh^\top] with the centered covariance Ck=Σk−μkμk⊤C_k = \Sigma_k - \mu_k\mu_k^\top. The results are shown below:

\hline & & \\;\text{Uncentered }(\Sigma_k)\\; & \quad\text{Centered }(C_k)\quad \\\\ \hline \text{Exp.} & \text{Diagnostic metric} & \llap{G_k}\quad\rlap{\Sigma_k} & \llap{G_k}\quad\rlap{C_k} \\\\ \hline & \text{Cross-client top-1 Dice} & \llap{0.972}\quad\rlap{0.998} & \llap{0.838}\quad\rlap{0.983} \\\\ \text{A} & \text{Within-client top-1 overlap }(G_k\text{ vs. }\Sigma_k\text{ or }C_k) & 0.887 & 0.191 \\\\ & \text{Held-out LOO action-energy capture (\\%)} & \llap{95.13}\quad\rlap{86.65} & \llap{78.37}\quad\rlap{18.27} \\\\ \hline & \text{Instruction tokens} & \llap{+0.215}\quad\rlap{+0.138} & \llap{+0.350}\quad\rlap{+0.312} \\\\ \text{B} & \text{Content tokens} & \llap{+0.165}\quad\rlap{+0.083} & \llap{+0.234}\quad\rlap{-0.067} \\\\ & \text{Decision tokens} & \llap{+0.218}\quad\rlap{+0.211} & \llap{+0.219}\quad\rlap{+0.286} \\\\ \hline \end{array}$$ A: RoBERTa-base on RTE. B: GPT-2 task-minus-domain contrast $S_{01}-S_{10}$. As shown in the table above, centering indeed changes the overall scale, as removing the mean term $\\|\Delta_k\mu_k\\|_2^2$ leaves $14.86\\%$ of the original action energy on RoBERTa and lowers the cross-client $G_k$ Dice. At the same time, the centered results show that $G_k$ agreement cannot be explained by the leading eigenvector of $C_k$. On RoBERTa, although the top eigenvectors of $C_k$ remain nearly identical across clients and $G_k$ also stays well aligned, their within-client overlap drops sharply from $0.887$ to $0.191$, indicating that they point in largely different directions. Consequently, other clients' centered $G_k$ directions capture $78.37\\%$ of the held-out centered action energy, compared with only $18.27\\%$ for $C_k$. A similar pattern holds on GPT-2, where the task-minus-domain contrasts of $G_k$ remain positive across all token regimes after centering, and on content tokens $G_k$ remains task-dominated ($+0.234$) whereas $C_k$ becomes domain-dominated ($-0.067$). Thus, while the shared mean inflates uncentered similarity, the learned action subspace retains genuine task-conditioned structure beyond the leading covariance direction. Finally, regarding S2, the transmitted directions and spectral weights summarize the adapter's action under each client's local input distribution. In the layer selector, regime-specific snapshots expose this structure separately for instruction, content, and decision tokens, allowing comparisons across clients and rounds that can reveal task or domain similarities. By contrast, sharing raw LoRA factors directly exposes the learned parameters, with both factors together determining the full adapter update. Quantifying the relative information leakage of these representations remains an open direction for future work, and we will include this discussion in the revised manuscript.
Reviewer_Sq5nOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes FedSAIL, a federated LoRA framework that shares input-aware action subspaces instead of directly averaging LoRA parameters. The authors show that parameter similarity can be heavily influenced by initialization, while input-aware action subspaces capture more consistent structures across clients. FedSAIL uses these shared subspaces to guide local training while preserving personalized adapters. Experiments on GLUE demonstrate improvements in accuracy and communication efficiency.

Critical strengths
  1. Interesting motivation. The paper provides an insightful observation that parameter similarity in federated LoRA may mainly come from shared initialization rather than actual shared knowledge.
  2. Well-designed framework. FedSAIL introduces input-aware subspace alignment while keeping local adapters personalized. The overall design is reasonable and easy to follow.
  3. Good writing. The paper is well-written and clearly organized. The motivation and methodology are easy to follow, and the figures help explain the main ideas.
Critical weaknesses

Major weaknesses:

  1. The main experiments only consider RoBERTa-large on GLUE with three clients. More experiments on larger language models are needed.
  2. The paper discusses FLoRA and FedIT but does not include them in the main comparisons.
  3. FedSAIL requires estimating input statistics, performing spectral decomposition, and applying additional regularization. The paper mainly reports communication costs but does not provide sufficient analysis of computation time or memory usage.
  4. The paper assumes that highly aligned action subspaces represent useful shared knowledge. However, high geometric similarity does not necessarily mean that collaboration along these directions improves downstream performance. More evidence is needed to establish this connection.
  5. I think some places, like the motivation and method, are overclaimed. For example, " Empirically, while parameter similarity vanishes, the leading right singular directions of this action matrix remain strongly aligned across clients." When can we judge vanishing? And do they have any observations or evidence that the right singular directions are strongly aligned?

Minor Weaknesses:

  1. The paper lacks a detailed sensitivity analysis of important hyperparameters.
Questions for authors
  1. How does FedSAIL perform on larger language models, such as LLaMA or Qwen?
  2. How sensitive is the layer-selection strategy to the manually selected threshold \tau?
  3. Why is a single global subspace appropriate when clients have conflicting tasks? Would clustering clients based on subspace similarity provide more effective collaboration?
9 comments
AuthorsOpenReview ↗

Response to Reviewer Sq5n: Major Weaknesses 1 and 2, and Question 1 (Part 1/7)

Thank you for these constructive suggestions. We have expanded our evaluation to include FLoRA and FedIT on GLUE, larger decoder-only language models (Qwen2.5-3B and Llama-3.2-3B) on generative tasks, and larger federations (10, 20, and 100 clients) under partial participation.

We first compare FedSAIL with FLoRA and FedIT following the same GLUE protocol as Table 2 in the main paper. As shown in the table below, while FedIT performs competitively and outperforms FLoRA under label skew, FedSAIL still maintains higher average accuracy than both baselines across all heterogeneity levels while requiring substantially less communication.

\hline \text{Validation accuracy on GLUE benchmark (\\%), 500 rounds, mean }\pm\text{ sample SD} \\\\ \hline \begin{array}{clccccccc} \alpha & \text{Method} & \text{RTE} & \text{SST-2} & \text{QNLI} & \text{MNLI-m} & \text{MNLI-mm} & \text{QQP} & \text{Avg.} \\\\ \hline 0.50 & \text{FLoRA} & 73.53 \pm 6.50 & 94.69 \pm 0.24 & 87.16 \pm 0.40 & 84.74 \pm 2.64 & 85.43 \pm 2.64 & 86.75 \pm 6.69 & 85.38 \pm 1.62 \\\\ 0.50 & \text{FedIT} & 79.54 \pm 4.98 & 95.87 \pm 0.70 & 92.23 \pm 0.26 & 88.61 \pm 1.64 & 88.68 \pm 1.88 & 88.99 \pm 5.01 & 88.99 \pm 1.68 \\\\ \hline 0.75 & \text{FLoRA} & 65.94 \pm 1.46 & 94.80 \pm 0.43 & 87.78 \pm 1.05 & 84.16 \pm 2.15 & 84.84 \pm 2.47 & 85.74 \pm 5.23 & 83.88 \pm 0.41 \\\\ 0.75 & \text{FedIT} & 79.18 \pm 1.99 & 95.60 \pm 0.07 & 91.60 \pm 0.50 & 88.16 \pm 1.43 & 88.43 \pm 1.45 & 88.19 \pm 3.89 & 88.53 \pm 0.30 \\\\ \hline 1.00 & \text{FLoRA} & 76.53 \pm 4.33 & 94.50 \pm 0.64 & 87.78 \pm 0.51 & 83.52 \pm 1.88 & 84.28 \pm 1.87 & 85.34 \pm 2.67 & 85.33 \pm 1.39 \\\\ 1.00 & \text{FedIT} & 77.86 \pm 4.03 & 95.53 \pm 0.83 & 91.74 \pm 0.39 & 87.87 \pm 1.23 & 88.04 \pm 1.43 & 87.76 \pm 2.18 & 88.13 \pm 1.13 \\\\ \end{array} \\\\ \hline \end{array}$$ Beyond encoder classification, we further evaluate larger decoder-only models on two generative benchmarks: (i) mathematical reasoning on GSM8K using Qwen2.5-3B across 5 IID clients, and (ii) code generation using both Qwen2.5-3B and Llama-3.2-3B trained on CodeSearchNet across 6 non-IID clients (one programming language per client) and evaluated on 158-task HumanEval benchmark using the Python client's adapter, as HumanEval is a Python benchmark (`Base` denotes the pretrained model before fine-tuning). While the overall gains on these generative benchmarks are modest, direct parameter averaging (FedAvg-LoRA and FedSA-LoRA) can suffer from cross-client interference and occasionally fall below the pretrained `Base` model. In contrast, FedSAIL improves over both the `Base` model and the competing baselines across all settings. $$\begin{array}{lcc} \hline & \text{GSM8K} & \text{HumanEval} \\\\ \hline \begin{array}{l} \text{Method} \\\\ \hline \text{Base} \\\\ \text{FedAvg-LoRA} \\\\ \text{FedSA-LoRA} \\\\ \hline \text{FedSAIL} \\\\ \end{array} & \begin{array}{c} \text{Qwen2.5-3B} \\\\ \hline 68.31 \\\\ 68.51 \pm 1.54 \\\\ 68.01 \pm 0.69 \\\\ \hline \mathbf{69.42 \pm 0.09} \\\\ \end{array} & \begin{array}{cc} \text{Qwen2.5-3B} & \text{Llama-3.2-3B} \\\\ \hline 40.51 & 27.85 \\\\ 41.14 & 26.58 \\\\ 39.24 & 27.85 \\\\ \hline \mathbf{41.77} & \mathbf{28.48} \\\\ \end{array} \\\\ \hline \end{array}$$ GSM8K: numerical exact match (%); HumanEval: pass rate (%). Finally, to assess scalability, we extend our RoBERTa-large evaluation on GLUE ($\alpha=0.75$) to 10, 20, and 100 clients over 1,000 rounds with a 0.3 client sampling rate per round. Rather than diminishing at scale, FedSAIL's advantage over the baselines becomes even more pronounced as the number of clients increases, since keeping both LoRA factors local avoids the instability of overwriting client adapters with partial averages under intermittent participation. $$\begin{array}{lccccccccccccccc} \hline \text{Method}&&\text{QNLI}&&&\text{SST-2}&&&\text{MNLI-m}&&&\text{MNLI-mm}&&&\text{Avg.}&\\\\ \hline &10&20&100&10&20&100&10&20&100&10&20&100&10&20&100 \\\\ &\text{clients}&\text{clients}&\text{clients}&\text{clients}&\text{clients}&\text{clients}&\text{clients}&\text{clients}&\text{clients}&\text{clients}&\text{clients}&\text{clients}&\text{clients}&\text{clients}&\text{clients} \\\\ \hline \text{Local-LoRA}&89.18&87.59&88.03&92.78&93.23&93.00&86.45&84.35&84.35&86.58&84.97&84.80&88.75&87.54&87.55 \\\\ \text{FFA-LoRA}&89.27&89.29&86.95&95.07&95.30&95.07&88.75&87.44&86.35&89.13&87.79&87.14&90.56&89.96&88.88 \\\\ \text{FedSA-LoRA}&90.92&92.17&90.01&94.84&94.72&94.50&90.23&87.29&84.20&90.99&87.85&84.52&91.74&90.51&88.30 \\\\ \hline \text{FedSAIL}&92.17&91.98&89.91&94.95&95.99&94.27&90.86&89.58&88.04&90.99&90.15&88.35&\mathbf{92.24}&\mathbf{91.93}&\mathbf{90.14} \\\\ \hline \end{array}$$
AuthorsOpenReview ↗

Response to Reviewer Sq5n: Major Weakness 3 (Part 2/7)

Thank you for pointing this out. To quantify the computational and memory overhead, we benchmarked the per-round wall-clock time and peak GPU memory of FedSAIL against the baselines using RoBERTa-large on QNLI (3 clients, r=8,s=1r=8, s=1) on an NVIDIA RTX 3090 GPU, the results are summarized below.

\hline \text{Method} & \text{Updates/client per round} & \text{Time/round (s)} & \text{Peak GPU memory (GiB)} \\\\ \hline \text{Local-LoRA} & 10 & 18.53\pm0.03 & 18.766 \\\\ \text{FedAvg-LoRA} & 10 & 18.63\pm0.18 & 18.766 \\\\ \text{FedSA-LoRA} & 10 & 18.66\pm0.20 & 18.766 \\\\ \text{FFA-LoRA} & 10 & 18.49\pm0.17 & 17.266 \\\\ \text{FedDPA-LoRA} & 10+10 & 39.20\pm0.38 & 19.493 \\\\ \hline \text{FedSAIL} & 10 & 19.38\pm0.11 & 18.955 \\\\ \hline \end{array}$$ Specifically, the costs of the three components in FedSAIL break down as follows: (1) **Input statistics estimation** is a one-time preprocessing step before training that takes $5.60\pm0.08$ s in total across all 3 clients and 48 modules (including $1.83$ s for matrix square roots), after which $\widehat{\Sigma}_k^{1/2}$ is kept fixed throughout training. (2) **Spectral decomposition and aggregation** avoid operating on full $q \times p$ matrices: each client performs SVD on the compact $r \times p$ matrix $\widehat{C}_k$ ($8 \times 1024$), and the server aggregates the weighted spectral factors via SVD on a $p \times (Ks)$ matrix ($1024 \times 3$). Across all 3 clients and 48 modules, the entire message-generation stage (including state loading and transfers) takes $0.360$ s per round, and server aggregation takes $0.038$ s. (3) **Subspace regularization** is evaluated as part of the full local training step, which takes $18.971$ s per round in total across the 3 clients—an incremental cost of only $\sim 0.44$ s per round compared with unregularized Local-LoRA ($18.53$ s). Overall, relative to FedSA-LoRA, our subspace-regularized approach increases per-round computation time by $3.82\\%$ ($+0.72$ s) and peak GPU memory by $1.01\\%$ ($\sim 193$ MiB). We will include this analysis in the revised manuscript.
AuthorsOpenReview ↗

Response to Reviewer Sq5n: Major Weakness 4 (Part 3/7)

Thank you for this insightful comment. We agree that high geometric alignment alone does not directly prove a gain in downstream accuracy in a complex model, and empirical support is needed. Nevertheless, a single-layer regression example explains why sharing can help.

Consider two clients whose rank-one action subspaces at a single layer share the same unknown unit direction vv. Each client estimates vv from its own data, obtaining v^k\hat v_k. By pooling v^1\hat v_1 and v^2\hat v_2, the clients can obtain a more accurate estimate v^fed\hat v_{\mathrm{fed}} while retaining their own output coefficients.

Specifically, denote the population least-squares risk on client kk by Lk(W)=Ek[∣Yk−Wh∣22]\mathcal L_k(W)=\mathbb{E}_k\bigl[\\|Y_k-Wh\\|_2^2\bigr] for a fresh input hh. Suppose Yk=Wkh+εkY_k=W_kh+\varepsilon_k, where Wk=BkAkW_k=B_kA_k, the noise has finite second moment, and Ek[εk∣h]=0\mathbb{E}_k[\varepsilon_k\mid h]=0. Then WkW_k minimizes this risk, with Lk(Wk)=Ek[∣εk∣22]\mathcal L_k(W_k)=\mathbb{E}_k\bigl[\\|\varepsilon_k\\|_2^2\bigr]. Writing Σk=Ek[hh⊤]\Sigma_k=\mathbb{E}_k[hh^\top], the excess-risk identity gives

=\mathbb{E}_k\bigl[\\|\bigl(W-W_k\bigr)h\\|_2^2\bigr] =\bigl\\|\bigl(W-W_k\bigr)\Sigma_k^{1/2}\bigr\\|_F^2.$$ Now assume $\Sigma_k$ is positive definite and write $z=\Sigma_k^{-1/2}h$. In these coordinates the true coefficient is the action matrix $G_k=W_k\Sigma_k^{1/2}=b_kv^\top$. Restricting prediction to a unit direction $u$ means that client $k$ uses only the scalar score $u^\top z$ and predicts $c\bigl(u^\top z\bigr)$ with its own output vector $c$. In the original input coordinates this is $W=cu^\top\Sigma_k^{-1/2}$. Substituting this coefficient into the excess-risk identity and minimizing over $c$ gives $$R_k(u) =\min_c\\,\mathbb{E}_k\bigl[\\|Y_k-c\bigl(u^\top z\bigr)\\|_2^2\bigr]-\mathcal L_k(W_k) =\min_c\\,\\|b_kv^\top-cu^\top\\|_F^2.$$ The minimizing coefficient is $c=G_ku=b_k\bigl(v^\top u\bigr)$, which gives $$R_k(u)=\\|b_k\\|_2^2\bigl[1-\bigl(u^\top v\bigr)^2\bigr].$$ Taking $u=\hat v_k$ gives the local excess risk $R_{\mathrm{loc},k}=R_k(\hat v_k)$, and taking $u=\hat v_{\mathrm{fed}}$ gives the pooled excess risk $R_{\mathrm{fed},k}=R_k(\hat v_{\mathrm{fed}})$. Thus, for a client with nonzero signal, pooling lowers prediction risk precisely when $\hat v_{\mathrm{fed}}$ is closer to the true line spanned by $v$ than $\hat v_k$ is. For example, suppose the two local estimates independently tilt from $v$ by $+\theta$ or $-\theta$ with equal probability in the same plane, where $0<\theta<\pi/4$. The leading direction of the equally weighted mean of their projectors recovers the true line when the tilts are opposite and coincides with the local estimate when they agree. Each event has probability $1/2$, so $\mathbb{E}\bigl[R_{\mathrm{fed},k}\bigr]=\tfrac12 R_{\mathrm{loc},k} =\tfrac12\\|b_k\\|_2^2\sin^2\theta$. Pooling therefore halves each client's expected excess prediction risk in this example.
Reviewer_pmghOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies what clients should share in Federated LoRA setting. It first shows that the similarity of LoRA A matrices across clients, which has been used to motivate sharing A, mostly comes from common initialization. Instead they propose FedSAIL, where clients keep both LoRA factors local and upload a small summary of main direction of their update. The server forms a shared subspace and clients are penalized if they update outside of it. They experiment with RoBERTa-large and provide cross-task benchmark suggest marginal gains in performance.

Critical strengths
  1. The diagnostics provided in Fig 2 and 3 are valuable and convincing how A-matrix similarity shrinks under independent initialization.

  2. The shared message is small and is rank independent, so comm savings are practical.

Critical weaknesses

My main concerns are about the empirical evidence provided:

W1. The results provided in Table 2. show small to moderate gains. The only real gains are shown for 2 out of 5 (RTE and QNLI) datasets.

W2. The gains have no statically significance. The gains are comparable or smaller than the reported standard deviation.

W3. The work seems to change two knobs simultaneously, clients share a direction and each client is penalized for updating outside it. The penalty alone can act as a regularizer and result in marginals gains. An experiment running FedSAIL with the same penalty but no communication could clear out if there are any gains strictly coming from the penalty acting as regularizer.

W4. The ablation provided in Table 4 is not supporting the claim. Sharing A gives about 20% less accuracy than plain LoRA from Table 2. A slightly less informative direction should not result in a 20% drop. Also the Appendix D says it is run with seed 123 only but reports std. deviation, how?

W5. Abstract mentions "Consistently improves" which is overstated as FedSAIL is not the best one in several columns despite having very low statistical value.

W6. No communication cost is reported for the baselines, although abstract claims significant reduction.

Questions for authors

Q1. What does FedSAIL score when each client uses its own subspace with no communication?

Q2. What exactly is the "LoRA" baseline here? Is it local-only training or averaging both factors?

5 comments
AuthorsOpenReview ↗

Response to Reviewer pmgh: W1, W2 and W5 (Part 1/5)

Thank you for pointing those out. We agree with W1 and W2 that the clearest gains in Table 2 are concentrated on RTE and QNLI rather than being uniform across all five tasks. Giving the three heterogeneity levels equal weight, FedSAIL gains 1.33 percentage points on RTE and 1.05 on QNLI over FedSA-LoRA, compared with 0.31 to 0.39 points on the other four splits. Also, the variability across seeds warrants care in interpreting the accuracy margins. Table 2 reports each method's variability across seeds, so the standard deviations alone do not establish whether the differences in mean accuracy are statistically significant, and we report these margins descriptively.

These concerns help us sharpen the scope of our contribution. FedSAIL has two components: (i) sharing the action subspace as an alternative to directly sharing the raw LoRA factor (AA), and (ii) using the resulting action subspace to construct a federated regularization penalty. The first component shows clear value across several settings, including robustness to independent LoRA initialization (Fig. 2), identifying layers with similar action subspaces (Fig. 3), and the Local-GkG_k control discussed later. The second component, however, does not yield uniform gains over all competitors.

There are several reasons for this: (1) Federated learning itself does not guarantee improvements on every task; (2) local refinement can already account for a substantial fraction of the gain, as suggested by the no-communication control in W3; and (3) for a strong pretrained backbone, client-specific fine-tuning may itself provide only limited improvement.

We therefore agree that FedSAIL's gains depend on the task and comparator, and we do not claim uniform dominance. As W5 suggested, the phrase “consistently improves” is too broad. Rather, the current results support a different advantage: FedSAIL achieves competitive, and often better, performance while substantially reducing communication through compact subspace sharing and layer-wise selection. It is also largely robust to independent LoRA initialization. We will revise the abstract and conclusion to describe competitive predictive performance with substantially lower communication cost in the evaluated settings.

We sincerely thank the reviewer for helping us sharpen this distinction and state the contribution more precisely.

AuthorsOpenReview ↗

Response to Reviewer pmgh: Q1 and W3 (Part 2/5)

Thank you for suggesting this insightful control. We compared three procedures on personalized GLUE. Pure local LoRA trains each client's adapter on its task loss alone. Local-GkG_k adds the residual penalty in Eq. (7), using the rank-one projector onto the leading right singular direction of the client's own input-weighted update, Gk=BkAkΣ^k1/2G_k=B_kA_k\widehat{\Sigma}_k^{1/2}. Both procedures use zero communication. FedSAIL uses the same form of penalty with a rank-one consensus projector estimated from the clients' spectral messages. The comparison between Local-GkG_k and pure local LoRA therefore addresses how much improvement is observed when the penalty uses only local information. The comparison between FedSAIL and Local-GkG_k addresses how much additional improvement is observed with the consensus direction.

\hline \text{Benchmark} & \text{Method} & \alpha=0.5 & \alpha=0.75 & \alpha=1.0 \\\\ \hline \text{RTE} & \text{Local-LoRA} & 78.58 \pm 9.32 & 75.81 \pm 3.13 & 78.70 \pm 2.17 \\\\ & \text{Local-}G_k & 79.66 \pm 6.46 & 77.74 \pm 3.35 & 80.14 \pm 3.31 \\\\[2pt] \text{SST-2} & \text{Local-LoRA} & 95.49 \pm 0.52 & 94.50 \pm 0.61 & 94.57 \pm 0.65 \\\\ & \text{Local-}G_k & 96.29 \pm 1.15 & 95.45 \pm 0.33 & 95.91 \pm 1.03 \\\\[2pt] \text{QNLI} & \text{Local-LoRA} & 90.57 \pm 0.34 & 90.66 \pm 0.44 & 90.67 \pm 0.74 \\\\ & \text{Local-}G_k & 92.75 \pm 0.54 & 92.93 \pm 0.41 & 93.31 \pm 0.54 \\\\[2pt] \text{MNLI-m} & \text{Local-LoRA} & 87.31 \pm 2.65 & 86.80 \pm 2.14 & 86.57 \pm 1.67 \\\\ & \text{Local-}G_k & 88.85 \pm 1.66 & 88.64 \pm 1.63 & 88.20 \pm 1.35 \\\\[2pt] \text{MNLI-mm} & \text{Local-LoRA} & 87.69 \pm 2.58 & 87.10 \pm 2.30 & 86.89 \pm 2.00 \\\\ & \text{Local-}G_k & 89.04 \pm 1.96 & 88.79 \pm 1.56 & 88.30 \pm 1.41 \\\\[2pt] \text{QQP} & \text{Local-LoRA} & 88.75 \pm 6.22 & 87.63 \pm 4.94 & 87.27 \pm 2.61 \\\\ & \text{Local-}G_k & 89.91 \pm 5.47 & 89.07 \pm 4.65 & 88.43 \pm 2.22 \\\\ \hline \text{Avg.} & \text{Local-LoRA} & 88.07 \pm 1.93 & 87.08 \pm 0.06 & 87.45 \pm 0.79 \\\\ & \text{Local-}G_k & 89.42 \pm 1.21 & 88.77 \pm 0.15 & 89.05 \pm 1.09 \\\\ & \text{FedSAIL (Table 2)} & 89.69 \pm 1.28 & 88.77 \pm 0.56 & 89.52 \pm 1.14 \\\\ \hline \end{array}$$ **Table R1:** GLUE validation accuracy (%) for Local-LoRA, Local-$G_k$, and FedSAIL. As shown in Table R1, Local-$G_k$ achieved higher mean validation accuracy than pure local LoRA on all six GLUE splits at each of the three heterogeneity levels. Its gains averaged over the six splits were 1.35, 1.69, and 1.60 percentage points at $\alpha=0.5, 0.75,$ and $1.0$, respectively. The corresponding differences between FedSAIL and Local-$G_k$ were +0.27, 0.00, and +0.47 points at the reported precision. Thus, in this experiment, the no-communication control supports the reviewer's intuition and clarifies how our method works. Regularizing toward each client's own action direction already recovered a large proportion of FedSAIL's observed gain over pure local LoRA on GLUE. The incremental gain from sharing was small on average, but its magnitude and sign varied across the evaluated settings. On RTE, for example, FedSAIL's mean accuracy was 1.81 percentage points higher than Local-$G_k$'s at $\alpha=1.0$.
AuthorsOpenReview ↗

Response to Reviewer pmgh: W4 (Part 3/5)

Thank you for this insightful observation. We agree that a slightly less informative direction should not cause an around 2020\\% accuracy drop. In the original Table 4, that drop occurred due to the reason that we varied only the shared message (AkA_k or Δk\Delta_k) while keeping the local regularizer fixed to the input-weighted action matrix G^k=ΔkΣ^k1/2\widehat{G}_k = \Delta_k \widehat{\Sigma}_k^{1/2}. Applying an unweighted projector P^\widehat{P} inside an Σ^k1/2\widehat{\Sigma}_k^{1/2}-weighted penalty causes a severe geometric mismatch, heavily penalizing updates along dominant input directions. To provide a fair, matched ablation, we re-evaluated the variants by matching each shared message with its corresponding local regularizer—specifically, pairing the AkA_k message with the AkA_k penalty β2∣Ak(Ip−P^)∣F2\frac{\beta}{2}\\|A_k(I_p - \widehat{P})\\|_F^2, and the Δk\Delta_k message with the Δk\Delta_k penalty β2∣Δk(Ip−P^)∣F2\frac{\beta}{2}\\|\Delta_k(I_p - \widehat{P})\\|_F^2:

\hline \text{Variant} & \begin{array}{c}\text{Shared} \\\\ \text{message}\end{array} & \begin{array}{c}\text{Local} \\\\ \text{regularizer}\end{array} & \begin{array}{c}\text{Input-} \\\\ \text{aware}\end{array} & \alpha=0.5 & \alpha=0.75 & \alpha=1.0 & \text{Overall} \\\\ \hline \text{Matched }A_k & A_k & A_k & \text{No} & 88.71_{\pm 1.83} & 88.01_{\pm 0.48} & 88.25_{\pm 1.31} & 88.33_{\pm 0.61} \\\\ \text{Matched }\Delta_k & \Delta_k & \Delta_k & \text{No} & 87.51_{\pm 3.06} & 87.01_{\pm 0.84} & 86.86_{\pm 1.17} & 87.13_{\pm 0.72} \\\\ \hline \text{FedSAIL} & \widehat{G}_k & \widehat{G}_k & \text{Yes} & \mathbf{89.69}_{\pm 1.28} & \mathbf{88.77}_{\pm 0.56} & \mathbf{89.52}_{\pm 1.14} & \mathbf{89.33}_{\pm 0.36} \\\\ \hline \end{array}$$ **Table R2:** Ablation study of the shared subspace and local regularizer on personalized GLUE (validation accuracy, %). As shown in Table R2, once the shared message and local regularizer are properly matched, the $\sim 20\\%$ drop disappears and the gap between variants becomes much smaller. Nevertheless, FedSAIL ($89.33\\%$) still maintains higher accuracy than both matched $A_k$ and matched $\Delta_k$ across all heterogeneity levels. We will update Table 4 with this matched comparison in the revised manuscript. These results are also consistent with our responses to W1, W3, and W5. On GLUE, consensus guidance yielded a modest average accuracy gain over local guidance, with effects varying by task. We also apologize for the text inconsistency in Appendix D. Table 4 in the main paper reports the mean and sample standard deviation across three seeds (123, 124, and 125), whereas Appendix D retained an earlier description and task-level breakdown for seed 123 only. We will update Appendix D to be fully consistent with the main text.

Comments on the paper

1 comment

Dear Area Chair,

We sincerely thank you and the four reviewers (pmgh, Sq5n, RqGd and WPTd) for the time and thoughtful reviews. With these additional experiments and analyses, we believe that our responses resolve most of the reviewers' concerns. Our main updates are summarized below:

  • More clients and partial participation: We scaled our GLUE experiments to 10, 20 and 100 clients with a 0.3 per-round sampling rate.
  • Decoder-only LLMs and generative tasks: We added experiments with Qwen2.5-3B and Llama-3.2-3B on mathematical reasoning (GSM8K) and code generation (CodeSearchNet / HumanEval).
  • Additional baselines and common initialization: We ran FFA-LoRA and FedSA-LoRA under their standard common initialization, clarified the FedAvg-LoRA baseline and added five more baselines (Local-LoRA, FLoRA, FedIT, FedRot-LoRA and LoRA-FAIR).
  • Subspace ablations and Σk\Sigma_k analysis: We updated Table 4 with a matched subspace-regularizer ablation, added a single-layer excess-risk analysis and ran controls on local regularization and the role of Σk\Sigma_k.
  • Communication, runtime and hyperparameter analyses: We reported explicit communication costs for all baselines, measured per-round runtime and peak GPU memory, as well as clarifying our hyperparameter choices.

We will include all of these new results and clarifications in the revised manuscript. Thank you very much for your time and work on our submission.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.