- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes an input-aware LoRA action matrix to identify shared subspaces between clients. FedSAIL guides local training by sharing spectral information while preserving personalized parameters. The experiments primarily validated accuracy and communication benefits under GLUE and cross-task settings.
Critical strengths
-
Distinguishing between parameter similarities arising from initialization and shared structures formed during training is a meaningful endeavor, and the experimental results offer valuable insights.
-
The design that decouples the shared subspace from local parameters is elegant, allowing for increased local adaptation capacity without increasing communication overhead.
Critical weaknesses
-
The baseline comparison is insufficient. The main experiments consistently employ independent initialization, whereas FFA-LoRA deviates from its standard "shared-frozen-A" configuration. The absence of a comparison under the standard setting makes it difficult to assess the method's advantage over the baseline operating in its intended mode.
-
The scope of validation is relatively narrow. The main experiment has only three clients per task, and the cross-task experiment has only eight clients, which is insufficient to support the effectiveness in general federated scenarios. The average accuracy gains are small, and there is a lack of validation on a larger scale and with the participation of some clients.
Questions for authors
-
Does FedSAIL maintain its advantages under the settings of joint initialization and standard FFA-LoRA?
-
Do performance and communication gains remain stable when the number of clients is increased and partial participation is employed?
2 comments
Response to Reviewer WPTd: W2 and Q2
Thank you for this important suggestion. To evaluate scalability and robustness under partial participation, we extended our RoBERTa-large experiments on GLUE () to federations of 10, 20, and 100 clients over 1,000 communication rounds, with a client sampling rate of 0.3 per round. Due to computational constraints during the rebuttal period, we first report validation accuracy from a single seed (seed 123) below, and will include full multi-seed means and standard deviations in the revised manuscript.
\begin{array}{lccccccccccccccc} \hline \text{Method} & & \text{QNLI} & & & \text{SST-2} & & & \text{MNLI-m} & & & \text{MNLI-mm} & & & \text{Avg.} & \\\\ \hline & 10 & 20 & 100 & 10 & 20 & 100 & 10 & 20 & 100 & 10 & 20 & 100 & 10 & 20 & 100 \\\\ & \text{clients} & \text{clients} & \text{clients} & \text{clients} & \text{clients} & \text{clients} & \text{clients} & \text{clients} & \text{clients} & \text{clients} & \text{clients} & \text{clients} & \text{clients} & \text{clients} & \text{clients} \\\\ \hline \text{Local-LoRA} & 89.18 & 87.59 & 88.03 & 92.78 & 93.23 & 93.00 & 86.45 & 84.35 & 84.35 & 86.58 & 84.97 & 84.80 & 88.75 & 87.54 & 87.55 \\\\ \text{FFA-LoRA} & 89.27 & 89.29 & 86.95 & 95.07 & 95.30 & 95.07 & 88.75 & 87.44 & 86.35 & 89.13 & 87.79 & 87.14 & 90.56 & 89.96 & 88.88 \\\\ \text{FedSA-LoRA} & 90.92 & 92.17 & 90.01 & 94.84 & 94.72 & 94.50 & 90.23 & 87.29 & 84.20 & 90.99 & 87.85 & 84.52 & 91.74 & 90.51 & 88.30 \\\\ \hline \text{FedSAIL} & 92.17 & 91.98 & 89.91 & 94.95 & 95.99 & 94.27 & 90.86 & 89.58 & 88.04 & 90.99 & 90.15 & 88.35 & 92.24 & 91.93 & 90.14 \\\\ \hline \end{array}$$ As shown in the results above, FedSAIL's performance advantage over the baselines becomes even more pronounced as the number of clients increases under partial participation, with its average accuracy margin over FedSA-LoRA widening from $+0.50\\%$ at 10 clients to $+1.42\\%$ at 20 clients and $+1.84\\%$ at 100 clients. With smaller local datasets and partial participation, directly overwriting local parameters with partial averages (as in FFA-LoRA and FedSA-LoRA) can destabilize local adaptation. In contrast, FedSAIL keeps both LoRA factors local and guides training via soft subspace regularization, allowing it to remain more stable against client drift. $$\small \begin{array}{lrrr} \hline \text{Method} & \text{10 clients} & \text{20 clients} & \text{100 clients} \\\\ \hline \text{FedSAIL (50 rounds warmup)} & 1.0447 & 2.0894 & 10.4472 \\\\ \text{FFA-LoRA} & 8.7891 & 17.5781 & 87.8906 \\\\ \text{FedSA-LoRA} & 8.7891 & 17.5781 & 87.8906 \\\\ \hline \text{Communication reduction} & 88.11\\% & 88.11\\% & 88.11\\% \\\\ \hline \end{array}$$ **Table 1:** Total bidirectional communication costs per run (GiB) over 1,000 rounds with a 0.3 sampling rate ($p=1024, r=8, s=1$, 4-byte entries). For communication efficiency, since each participating client's per-round payload is independent of the federation size, the total communication volume scales linearly with the number of sampled clients per round. As shown in Table 1, FedSAIL consistently reduces total communication cost by 88.11% across all client scales compared to both baselines.
Response to Reviewer WPTd: W1 and Q1
Thank you for pointing this out. We agree that evaluating FFA-LoRA under its standard configuration, where clients share a common frozen initialization of A, is essential for a fair comparison. Following your suggestion, we evaluated FFA-LoRA under common initialization across the GLUE benchmark and also included the corresponding results for FedSA-LoRA, as reported below.
\hline \text{Validation accuracy on GLUE benchmark (\\%), mean } \pm \text{ sample SD} \\\\ \hline \begin{array}{cllccccccc} \alpha & \text{Method} & \text{Init.} & \text{RTE} & \text{SST-2} & \text{QNLI} & \text{MNLI-m} & \text{MNLI-mm} & \text{QQP} & \text{Avg.} \\\\ \hline 0.50 & \text{FedSA-LoRA} & \text{Ind.} & 79.90_{\pm 8.61} & 95.95_{\pm 1.13} & 91.76_{\pm 0.61} & 88.56_{\pm 1.72} & 88.92_{\pm 1.76} & 89.53_{\pm 5.63} & 89.10_{\pm 1.60} \\\\ & \text{FedSA-LoRA} & \text{Common} & 80.87_{\pm 7.51} & 96.14_{\pm 1.36} & 92.20_{\pm 0.56} & 88.81_{\pm 1.70} & 88.88_{\pm 1.72} & 89.54_{\pm 5.68} & 89.40_{\pm 1.34} \\\\ & \text{FFA-LoRA} & \text{Ind.} & 70.04_{\pm 5.91} & 90.75_{\pm 1.78} & 79.46_{\pm 2.06} & 65.46_{\pm 9.63} & 66.65_{\pm 9.11} & 81.28_{\pm 11.53} & 75.61_{\pm 3.27} \\\\ & \text{FFA-LoRA} & \text{Common} & 70.88_{\pm 3.47} & 95.15_{\pm 1.03} & 84.86_{\pm 2.21} & 84.02_{\pm 3.63} & 84.83_{\pm 3.71} & 86.78_{\pm 6.78} & 84.42_{\pm 1.17} \\\\ \hline 0.75 & \text{FedSA-LoRA} & \text{Ind.} & 76.65_{\pm 3.86} & 95.41_{\pm 0.41} & 92.34_{\pm 0.29} & 88.22_{\pm 1.58} & 88.42_{\pm 1.75} & 88.70_{\pm 4.87} & 88.29_{\pm 0.39} \\\\ & \text{FedSA-LoRA} & \text{Common} & 77.14_{\pm 2.94} & 95.72_{\pm 0.59} & 92.89_{\pm 0.42} & 88.39_{\pm 1.58} & 88.68_{\pm 1.66} & 89.00_{\pm 4.66} & 88.63_{\pm 0.13} \\\\ & \text{FFA-LoRA} & \text{Ind.} & 61.13_{\pm 3.47} & 89.22_{\pm 2.81} & 81.77_{\pm 2.42} & 62.33_{\pm 10.84} & 63.60_{\pm 10.45} & 81.53_{\pm 8.88} & 73.26_{\pm 1.81} \\\\ & \text{FFA-LoRA} & \text{Common} & 64.02_{\pm 2.66} & 94.69_{\pm 0.24} & 86.54_{\pm 0.82} & 83.58_{\pm 2.72} & 84.45_{\pm 3.00} & 86.05_{\pm 5.41} & 83.22_{\pm 0.96} \\\\ \hline 1.00 & \text{FedSA-LoRA} & \text{Ind.} & 79.54_{\pm 4.35} & 95.83_{\pm 0.81} & 92.32_{\pm 0.66} & 88.10_{\pm 1.52} & 88.17_{\pm 1.60} & 88.18_{\pm 2.14} & 88.69_{\pm 1.31} \\\\ & \text{FedSA-LoRA} & \text{Common} & 79.78_{\pm 3.08} & 95.83_{\pm 1.03} & 92.71_{\pm 0.42} & 88.04_{\pm 1.48} & 88.17_{\pm 1.55} & 88.29_{\pm 2.15} & 88.81_{\pm 1.06} \\\\ & \text{FFA-LoRA} & \text{Ind.} & 76.29_{\pm 4.22} & 87.88_{\pm 0.98} & 80.08_{\pm 1.36} & 61.48_{\pm 11.53} & 62.48_{\pm 10.99} & 81.69_{\pm 4.80} & 74.98_{\pm 4.05} \\\\ & \text{FFA-LoRA} & \text{Common} & 76.41_{\pm 4.34} & 94.76_{\pm 0.98} & 87.39_{\pm 1.21} & 83.18_{\pm 2.92} & 83.84_{\pm 2.76} & 85.25_{\pm 2.62} & 85.14_{\pm 1.92} \\\\ \end{array} \\\\ \hline \end{array}$$ As shown in the table above, indeed, adopting common initialization improves FFA-LoRA's average accuracy by approximately $9\text{-}10$ percentage points across the GLUE benchmark, as a shared frozen $A$ provides a consistent projection basis for aggregating $B_k$. Motivated by this observation, we also examined the impact of initialization on FedSA-LoRA. Although the official implementation of FedSA-LoRA uses common initialization, our results indicate that its performance is largely insensitive to the initialization scheme, since $A_k$ is actively trained and synchronized in every round. Overall, even after updating the baselines to their common-initialization results, our FedSAIL approach still maintains its performance advantage while substantially reducing communication costs.