acceptodds
Under review as a conference paper at ICLR 2027

ProLoRA: Bidirectional Communication-Efficient Federated Fine-Tuning with LoRA-Aware Error Feedback

Abstract

Federated fine-tuning of large language models (LLMs) using low-rank adaptation (LoRA) addresses data scarcity while preserving data privacy. However, the frequent exchange of LoRA updates leads to bidirectional communication bottlenecks. Existing solutions either reduce communication by restricting the trainable LoRA factors, leading to performance degradation, or compress LoRA updates, resulting in high downlink communication overhead or additional server-side compression. To overcome these challenges, we propose ProLoRA by designing novel and efficient random orthogonal projection (ROP) compressor and LoRA-aware error feedback (LA-EF), which enables efficient bidirectional communication while maintaining high model performance. Specifically, ROP enables direct aggregation of compressed LoRA updates through its linearity, while LoRA-aware error feedback exploits shared LoRA geometry to regulate historical compression residuals and improve model recovery. We theoretically prove that ProLoRA achieves the same convergence order as full-precision FedAvg. We conduct extensive experiments on multiple datasets and LLMs to demonstrate that ProLoRA reduces total communication costs by up to 96.9% without sacrificing model performance when compared with state-of-the-art (SOTA) solutions.

Primary area:
optimization
Keywords:
Federated Learning, Low-Rank Adaptation, Communication Efficiency

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_7HJbOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes ProLoRA, a communication-efficient federated LoRA fine-tuning framework combining Random Orthogonal Projection and LoRA-Aware Error Feedback. ROP enables direct aggregation and broadcasting of compressed LoRA updates, while LA-EF exploits the coupling between LoRA factors to regulate historical compression residuals. The authors provide convergence analysis and evaluate the method across multiple datasets and language models, demonstrating substantial communication reduction while maintaining competitive performance.

Critical strengths

The paper addresses bidirectional communication overhead in federated LoRA fine-tuning, an important challenge when communication resources are limited. The shared orthogonal projection design allows for the compression of update parameters, thereby reducing communication costs.

Critical weaknesses
  1. Limited methodological novelty and insufficient justification of LA-EF. The core ROP design builds on well-established random projection and linear aggregation properties, while error feedback is also a widely studied technique. Thus, the main methodological novelty appears to lie in LA-EF. The article also lacks clarity regarding the design rationale and details of the LA-EF.
  2. Aggregation bias still exists, which affects federated training performance. When ProLoRA is aggregated in the cloud, it still behaves similarly to averaging over the same A or B factors. Due to the bilinear nature of LoRA, this can lead to aggregation bias issues. LA-EF primarily compensates for compression errors, rather than factor aggregation errors.
Questions for authors
  1. Under the same compression ratio and optimization budget, what advantages does LA-EF offer over simpler residual damping or adaptive filtering strategies?
  2. LA-EF attenuates historical compression residuals using the Gram matrices of LoRA factors, potentially suppressing useful accumulated information. Can the authors provide stronger theoretical justification for this trade-off between noise reduction and information preservation?
5 comments
AuthorsOpenReview ↗

Author Response to Reviewer 7HJb (Part 2/2)

3. Advantages of LA-EF over Simple Residual Damping and Alternative Filtering Strategies (W1, Q1)

We have added controlled ablation experiments in Appendix F.1 of the revised manuscript. All methods use the same ROP compressor and are evaluated on MNLI with 10 clients, 300 communication rounds, α=0.5\alpha=0.5, and q=1/16q=1/16. Fixed scalar damping uses c=(1+q)−1c=(1+q)^{-1}, while tuned scalar damping selects the best coefficient from {0.5,0.8,(1+q)−1,0.98}\{0.5,0.8,(1+q)^{-1},0.98\}. Diagonal Gram filtering uses only the diagonal entries of the Gram matrix. All ROP variants use the same expanded global learning-rate search grid {0.125,0.25,0.5,1,1.5,2}\{0.125,0.25,0.5,1,1.5,2\}.

MethodMNLI Accuracy (%)
FedIT (Uncompressed)84.76 ± 0.21
ROP (Without EF)76.80 ± 2.72
ROP + Standard EF73.27 ± 6.52
ROP + Fixed Scalar Damping77.58 ± 6.92
ROP + Tuned Scalar Damping80.35 ± 1.02
ROP + Diagonal Gram Filtering78.54 ± 7.23
ROP + LA-EF (ProLoRA)84.34 ± 0.49

Simple scalar damping can indeed mitigate the negative effects of standard EF, but it cannot match LA-EF. LA-EF outperforms the best tuned scalar damping strategy by 3.99% and diagonal Gram filtering by 5.80%, while remaining only 0.42% below uncompressed FedIT. These results demonstrate that LA-EF's performance gains do not arise solely from uniformly attenuating historical residuals, and that the complete LoRA geometry provides additional benefits.

4. Aggregation Bias Arising from the Two-Factor Structure of LoRA (W2)

We agree that separately aggregating the two LoRA factors is generally not equivalent to directly aggregating the effective weights. Let Bˉ=∑ipiBi\bar{\mathbf B}=\sum_i p_i\mathbf B_i and Aˉ=∑ipiAi\bar{\mathbf A}=\sum_i p_i\mathbf A_i. Then,

∑ipiBiAi−BˉAˉ=∑ipi(Bi−Bˉ)(Ai−Aˉ).\sum_i p_i\mathbf B_i\mathbf A_i-\bar{\mathbf B}\bar{\mathbf A}=\sum_i p_i(\mathbf B_i-\bar{\mathbf B})(\mathbf A_i-\bar{\mathbf A}).

However, the two-factor aggregation bias and ROP's communication compression error are distinct problems. The former may exist even without communication compression, while LA-EF addresses only the latter without modifying the server-side aggregation rule.

To evaluate its practical impact, we conducted an additional comparison between FedIT and FedEx-LoRA (Singhal et al., ACL 2025). FedEx-LoRA achieves exact aggregation of effective weight updates by introducing a correction residual. We evaluate both methods on MNLI with 10 clients, 300 communication rounds, and α=0.5\alpha=0.5.

MethodMNLI Accuracy (%)
FedIT (Separate Factor Aggregation)84.76 ± 0.21
FedEx-LoRA (Exact Effective Update Aggregation)84.68 ± 0.22

Under this setting, exact aggregation does not yield a noticeable accuracy improvement. This does not imply that aggregation bias is unimportant in other settings. ProLoRA focuses on bidirectional communication compression and historical residual feedback rather than eliminating two-factor aggregation bias.

Singhal, Ponkshe, and Vepakomma. “FedEx-LoRA: Exact Aggregation for Federated and Efficient Fine-Tuning of Large Language Models.” ACL 2025.

AuthorsOpenReview ↗

Author Response to Reviewer 7HJb (Part 1/2)

We thank the reviewer for the careful evaluation of our work and for recognizing the value of bidirectional communication compression. For clarity, W1–W2 refer to Critical Weaknesses, while Q1–Q2 refer to Questions for Authors.

1. Methodological Novelty and Design Motivation of ROP and LA-EF (W1)

We believe that the novelty of ProLoRA lies in the joint design of ROP and LA-EF, rather than a simple combination of existing random projection and error feedback techniques. ROP constructs a compact bidirectional communication representation tailored to the two-factor structure of LoRA, enabling the server to directly aggregate and broadcast compressed updates without additional compression while keeping both LoRA factors fully trainable locally. We have expanded the related-work discussion in Section 2 and added direct comparisons with FetchSGD and FedKRSO in Appendix F.7 of the revised manuscript.

More importantly, we find that directly combining ROP with standard EF may impair model optimization. To explain this phenomenon, we have added a stochastic quadratic model analysis in Section 4.1 and Appendix B.1, showing that when both methods are mean-square stable and persistent perturbation is nonzero, standard EF introduces an additional historical-feedback penalty:

ΠEF=2(1−q)qγ2.\Pi_{\mathrm{EF}}=\frac{2(1-q)}{q}\gamma^2.

We further reveal in Appendix B.2 that the Gram spectrum of the partner LoRA factor makes historical-feedback costs direction-dependent. Motivated by this finding, LA-EF regulates historical residuals using the global LoRA geometry while preserving ROP's compact communication representation without additional communication overhead. Its quadratic optimization objective and closed-form solution have been added to Section 4.1 and Appendix B.3.

2. Theoretical Justification of LA-EF and the Trade-off Between Residual Retention and Noise Suppression (W1, Q2)

LA-EF aims to balance historical information retention and projection noise suppression, rather than simply reducing historical residuals. We have added the following theoretical analyses in Section 4.1 and Appendix B.3 of the revised manuscript.

For historical residuals Y=(YB,YA)\mathbf Y=(\mathbf Y_B,\mathbf Y_A), when the two factors use independent random orthogonal projections with the same retention ratio, the first-order projection noise in the effective LoRA weight space satisfies

ECpt[∥Jt((Cpt−qI)Y)∥2]=q(1−q)(∥YBAt∥2+∥BtYA∥2).\mathbb E_{\mathcal C_p^t}[\|J_t((\mathcal C_p^t-q\mathcal I)\mathbf Y)\|^2]=q(1-q)(\|\mathbf Y_B\mathbf A^t\|^2+\|\mathbf B^t\mathbf Y_A\|^2).

This result reveals the dependence of projection noise on the geometry of the partner factors. Minimizing this noise alone yields the trivial zero-residual solution. Therefore, we formulate a strictly convex quadratic objective that jointly controls historical information loss and direction-dependent projection noise:

min⁡YB,YA 12(∥YB−eB∥2+∥YA−eA∥2)+q2(∥YBAt∥2μBt+∥BtYA∥2μAt).\min_{\mathbf Y_B,\mathbf Y_A}\ \frac12(\|\mathbf Y_B-\mathbf e_B\|^2+\|\mathbf Y_A-\mathbf e_A\|^2)+\frac q2\left(\frac{\|\mathbf Y_B\mathbf A^t\|^2}{\mu_B^t}+\frac{\|\mathbf B^t\mathbf Y_A\|^2}{\mu_A^t}\right).

Here, μbt=tr⁡(Gbt)/r\mu_b^t=\operatorname{tr}(\mathbf G_b^t)/r. For nonzero Gram matrices, this objective admits a unique closed-form solution:

Hbt=(Ir+qrGbttr⁡(Gbt))−1.\mathbf H_b^t=\left(\mathbf I_r+q\frac{r\mathbf G_b^t}{\operatorname{tr}(\mathbf G_b^t)}\right)^{-1}.

For zero Gram matrices, we use an isotropic filter as a fallback.

Furthermore, we establish a direction-dependent theoretical advantage of LA-EF over scalar damping under the same residual norm (Appendix B.3):

ciso2∥eBAt∥2−∥eBHBtAt∥2=−∥eB∥2Cov⁡w(λj,hj2)≥0.c_{\mathrm{iso}}^2\|\mathbf e_B\mathbf A^t\|^2-\|\mathbf e_B\mathbf H_B^t\mathbf A^t\|^2=-\|\mathbf e_B\|^2\operatorname{Cov}_w(\lambda_j,h_j^2)\ge0.

Here, cisoc_{\mathrm{iso}} satisfies ∥cisoeB∥=∥eBHBt∥\|c_{\mathrm{iso}}\mathbf e_B\|=\|\mathbf e_B\mathbf H_B^t\|. This result shows that, under the same residual norm, LA-EF produces no greater partner-weighted residual effect than uniform scalar damping, demonstrating the theoretical value of direction-dependent filtering.

Dear Reviewer 7HJb,

Thank you for your valuable feedback on our paper. As only one day remains in the discussion period, we would greatly appreciate it if you could review our rebuttal and revised manuscript and share any remaining questions or concerns. We would be happy to provide further clarification and engage in discussion.

Over the past few days, we have engaged in extensive discussions with other reviewers, particularly Reviewer K7M4. In addition to our rebuttal and revised manuscript, we sincerely invite you to review these discussions, which provide further clarifications on several important aspects of our work. We believe these clarifications and discussions may help provide a more comprehensive understanding of our contributions. We would be very grateful if you could reconsider your assessment in light of these updates.

Thank you again for your time and consideration.

Reviewer_grNCOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ProLoRA for reducing both uplink and downlink communication in federated LoRA fine-tuning. Clients use shared random orthogonal projections to compress updates to both LoRA factors, allowing the server to aggregate and broadcast the compressed representations directly. A LoRA-aware error-feedback mechanism filters historical residuals using the Gram matrices of the partner factors. The paper provides a non-convex stationarity bound with an O(T−1/2)\mathcal{O}(T^{-1/2}) rate for fixed compression settings under its stated assumptions. Experiments compare communication and performance across GLUE tasks and larger language models, with additional ablations and resource measurements.

Critical strengths
  1. The communication protocol addresses a concrete limitation of independently sparsified client updates: aggregation can make the downlink representation dense. Shared linear projections preserve a compact representation through aggregation while allowing both LoRA factors to remain trainable. The experiments examine equal rounds, equal communication budgets, and target accuracy, which gives a useful view of the trade-off. At q=1/16q = 1/16, the reported communication reduction relative to FedIT is substantial, with generally close performance in the main settings.

  2. The residual-filtering component is supported by substantial empirical and analytical work. Figures 5, 7, and 8 show large improvements over standard error feedback in several compressed, heterogeneous settings. The convergence analysis explicitly handles the changing factor geometry and filtered residuals. I also appreciate the larger-model experiments and the timing and memory breakdowns in Appendix E, including the separate reporting of CPU storage.

Critical weaknesses
  1. The experiments do not yet isolate the benefit of the LoRA geometry from the benefit of damping historical residuals. The main ablation compares LA-EF with standard, unfiltered EF. Since the proposed filter attenuates residuals, scalar filters HB=HA=cIrH_B = H_A = c I_r, with 0<c<10 < c < 1, could also reduce the residual-induced projection noise. The noise reduction in Appendix B therefore does not establish that the directional information in the Gram matrices is responsible for the performance recovery. This distinction matters because geometry-aware feedback is a central methodological contribution.

  2. The claim of preserving performance needs qualification. The main results are encouraging, but Appendix E.4, Table 6 reports 79.84±2.8279.84 \pm 2.82 on MNLI at Dirichlet α=0.3\alpha = 0.3, compared with 83.74±0.1283.74 \pm 0.12 for FedIT: an observed mean gap of 3.90 percentage points. This is an important exception in a setting directly relevant to the paper's motivation. The headline 96.9% communication saving also corresponds to q=1/32q = 1/32 relative to uncompressed communication, rather than a uniform saving over the strongest compressed baseline at matched accuracy. The paper should distinguish these comparisons more clearly.

Questions for authors
  1. Can you compare LA-EF with scalar-damped EF under the same projection, training, and tuning settings? A natural control is

    HB=HA=11+qIr,H_B = H_A = \frac{1}{1+q} I_r,

    corresponding to the isotropic case of the proposed filter, alongside a validation-tuned scalar control HB=HA=cIrH_B = H_A = c I_r.

  2. Were the learning rates for ROP+EF selected independently using the same search procedure as ProLoRA? This would help interpret the large recovery differences in Figures 7 and 8.

  3. Can you explain the remaining MNLI degradation at α=0.3\alpha = 0.3 and state more precisely the settings in which performance preservation is supported?

  4. The convergence proof assumes the same retention ratio qq on both factor sides. For rectangular layers with m≠nm \neq n, how does the implementation choose the projection dimensions, given that the main algorithm uses a single ss? With that choice, the two retention ratios would be s/ms/m and s/ns/n, respectively.

5 comments
AuthorsOpenReview ↗

Author Response to Reviewer grNC (Part 1/2)

We thank the reviewer for the careful evaluation of our work and for recognizing ProLoRA's bidirectional communication design and LA-EF's theoretical and experimental results. For clarity, W1–W2 refer to Critical Weaknesses, while Q1–Q4 refer to Questions for Authors.

1. Advantages of LoRA Geometry-Aware Feedback over Scalar Damping and Ablation Validation (W1, Q1)

We thank the reviewer for highlighting the need to distinguish the contribution of LoRA geometry from simple residual damping. We agree that comparing LA-EF only with standard EF is insufficient to establish the necessity of geometry-aware residual regulation. We have therefore added controlled ablations with fixed scalar damping, tuned scalar damping, diagonal Gram filtering, and ROP without EF in Appendix F.1 of the revised manuscript.

First, we compare all methods under the same compressor, training configuration, and tuning conditions. All methods use ROP and are evaluated on RoBERTa-base/MNLI with 10 clients, 300 communication rounds, Dirichlet parameter α=0.5\alpha=0.5, and compression ratio q=1/16q=1/16. Fixed scalar damping uses the reviewer's suggested HB=HA=(1+q)−1IrH_B=H_A=(1+q)^{-1}I_r, while tuned scalar damping selects the best coefficient from {0.5,0.8,(1+q)−1,0.98}\{0.5,0.8,(1+q)^{-1},0.98\}. We also compare diagonal Gram filtering, which uses only Gram diagonal entries, with full LA-EF. For fairness, all ROP variants use the same expanded global learning-rate grid {0.125,0.25,0.5,1,1.5,2}\{0.125,0.25,0.5,1,1.5,2\}. The new results are:

MethodMNLI Accuracy (%)
FedIT (Uncompressed Reference)84.76 ± 0.21
ROP (Without EF)76.80 ± 2.72
ROP + Standard EF73.27 ± 6.52
ROP + Fixed Scalar Damping77.58 ± 6.92
ROP + Tuned Scalar Damping80.35 ± 1.02
ROP + Diagonal Gram Filtering78.54 ± 7.23
ROP + LA-EF (ProLoRA)84.34 ± 0.49

The results show that simple damping improves performance but cannot match full LA-EF. Standard EF achieves only 73.27%, below the 76.80% without EF. Fixed and tuned scalar damping achieve 77.58% and 80.35%, respectively. In contrast, LA-EF achieves 84.34%, outperforming the best scalar damping by 3.99% and diagonal Gram filtering by 5.80%, while remaining only 0.42% below uncompressed FedIT. Thus, LA-EF benefits not only from reducing feedback strength but also from rank-direction correlations captured by the full Gram matrix.

We have also added a direction-dependent theoretical analysis in Appendix B.3. At equal residual norms, with cisoc_{\mathrm{iso}} satisfying ∥cisoeB∥=∥eBHBt∥\|c_{\mathrm{iso}}\mathbf e_B\|=\|\mathbf e_B\mathbf H_B^t\|, we prove:

ciso2∥eBAt∥2−∥eBHBtAt∥2=−∥eB∥2Cov⁡w(λj,hj2)≥0.c_{\mathrm{iso}}^2\|\mathbf e_B\mathbf A^t\|^2-\|\mathbf e_B\mathbf H_B^t\mathbf A^t\|^2=-\|\mathbf e_B\|^2\operatorname{Cov}_w(\lambda_j,h_j^2)\ge0.

Under the same residual norm, LA-EF has no greater partner-weighted residual effect on the effective LoRA weights than uniform scalar damping. Unlike scalar damping, LA-EF applies direction-dependent feedback based on the Gram eigenvalues. This theoretical property is consistent with our ablation results.

2. Fairness of Learning-Rate Tuning for Standard EF and LA-EF (Q2)

We thank the reviewer for raising the issue of learning-rate tuning fairness. We agree that different feedback strategies may favor different learning rates and should be tuned sufficiently and fairly.

In the original experiments, all methods used the same global learning-rate search grid {1,1.5,2}\{1,1.5,2\} and independently selected their best configurations. However, we subsequently found that better learning rates for standard EF may lie outside this initial grid, which therefore did not fully reflect its best performance.

We uniformly added three smaller learning rates {0.125,0.25,0.5}\{0.125,0.25,0.5\} for every ROP variant, expanding the grid to {0.125,0.25,0.5,1,1.5,2}\{0.125,0.25,0.5,1,1.5,2\}, and independently retuned all variants. Based on these results, we updated Figure 5 in Section 6.4 and Figures 7 and 8 in Appendix F.1 of the revised manuscript and added corresponding ablations.

The updated results confirm LA-EF's substantial advantage. On MNLI with α=0.5\alpha=0.5, q=1/16q=1/16, and 300 rounds, standard EF achieves 73.27 ± 6.52%, below ROP without EF (76.80 ± 2.72%), whereas LA-EF reaches 84.34 ± 0.49%, an 11.07% improvement over standard EF. Under stronger non-IID data with α=0.3\alpha=0.3, LA-EF improves MNLI and QNLI accuracy from 54.37% and 80.29% with standard EF to 79.84% and 90.62%, respectively.

Thus, even after expanding the same search grid and independently retuning all methods, standard EF may still impair ROP's performance, while LA-EF maintains a clear recovery advantage. The updated Figures 5, 7, and 8 and the ablations in Appendix F.1 support this conclusion.

AuthorsOpenReview ↗

Author Response to Reviewer grNC (Part 2/2)

3. Performance Limitations under Strong Non-IID Data and Clarification of Communication Savings (W2, Q3)

We thank the reviewer for identifying performance degradation under strong data heterogeneity and the need to clarify the baseline for communication savings. We agree that ProLoRA's ability to preserve accuracy depends on data heterogeneity and compression ratio, so the scope of our claims should be explicit.

First, we acknowledge that ProLoRA can still lose accuracy under stronger data heterogeneity. At Dirichlet α=0.3\alpha=0.3, ProLoRA achieves 79.84% on MNLI, a 3.90% gap from uncompressed FedIT's 83.74%, while the QNLI gap is 0.52%. The results are:

MethodMNLI (%)QNLI (%)
FedIT83.74 ± 0.1291.14 ± 0.86
RoLoRA78.96 ± 4.1189.67 ± 1.38
EcoLoRA78.84 ± 0.5389.91 ± 0.64
ProLoRA79.84 ± 2.8290.62 ± 1.10

This gap may reflect the combined effects of stronger data heterogeneity and compression. Heterogeneity may increase discrepancies among client updates, while ROP's finite-dimensional projections still introduce information loss. Although LA-EF improves historical residual feedback, it cannot completely remove these effects.

Notably, ProLoRA still achieves the highest mean accuracy among the evaluated communication-efficient methods at α=0.3\alpha=0.3, while reducing total communication by 93.75% relative to FedIT. We explicitly report the 3.90% MNLI gap in Appendix F.4 of the revised manuscript and qualify our performance-preservation claims accordingly.

Second, we clarify the reference baseline for the reported 96.9% communication savings. This figure means 96.9% lower total uplink and downlink communication than uncompressed FedIT at q=1/32q=1/32. It is not a reduction relative to the strongest compressed baseline and does not apply to every compression ratio. At the main setting q=1/16q=1/16, the corresponding saving relative to FedIT is 93.75%. We have clarified the reference baseline in Section 6 and reported performance recovery under different compression ratios in Appendix F.1.

Accordingly, our claim is competitive accuracy in the main experimental settings alongside substantial bidirectional communication savings, not lossless performance under every data heterogeneity or compression setting.

4. Projection Dimension Selection and Convergence Guarantees for Rectangular LoRA Layers (Q4)

We thank the reviewer for noting the relationship between projection dimensions and compression ratios for rectangular LoRA layers. If both factors use the same projection dimension ss with m≠nm\ne n, their ratios are qB=s/mq_B=s/m and qA=s/nq_A=s/n, so a common-ratio convergence result does not directly apply. We therefore added a rectangular-layer analysis and Theorem 2 in Appendix E.1 of the revised manuscript.

First, rectangular factors need not share the same projection dimension. For B∈Rm×r\mathbf B\in\mathbb R^{m\times r} and A∈Rr×n\mathbf A\in\mathbb R^{r\times n}, ROP can choose sBs_B and sAs_A independently, with

qB=sBm,qA=sAn.q_B=\frac{s_B}{m},\qquad q_A=\frac{s_A}{n}.

Setting sB/m=sA/n=qs_B/m=s_A/n=q gives both factors the same compression ratio. We use this strategy in large-model experiments, so the original common-ratio convergence guarantee remains applicable. The compressed factor representations have dimensions sB×rs_B\times r and r×sAr\times s_A, and the server can still directly aggregate and broadcast them in the compressed domain.

Second, we also analyze unequal factor-specific compression ratios. A common projection dimension or integer dimension constraints may yield qB≠qAq_B\ne q_A. Appendix E.1 extends the convergence analysis to this case. Let q−=min⁡bqbq_- =\min_b q_b and q+=max⁡bqbq_+ =\max_b q_b, with factor-specific residual filter Hbt=(Ir+qbSbt)−1\mathbf H_b^t=(\mathbf I_r+q_b\mathbf S_b^t)^{-1}.

Under the corresponding smoothness, gradient-variance, data-heterogeneity, and learning-rate assumptions, Theorem 2 establishes:

1T∑t=1TE[∥∇F(θt)∥2]=O(q−ΔF+q+σ2q−2TEN+σ2+Eδ2q−ET+q+(1−q−)σ2q−4T).\frac1T\sum_{t=1}^T\mathbb E[\|\nabla F(\boldsymbol{\theta}^t)\|^2]=\mathcal O\left(\frac{q_-\Delta_F+\sqrt{q_+}\sigma^2}{q_-^2\sqrt{TEN}}+\frac{\sigma^2+E\delta^2}{q_-ET}+\frac{q_+(1-q_-)\sigma^2}{q_-^4T}\right).

For fixed positive compression ratios, ProLoRA retains an O(T−1/2)\mathcal O(T^{-1/2}) convergence order even when the factors use different ratios. When q−=q+=qq_-=q_+=q, this reduces to the original convergence bound. Unequal ratios affect the bound's constants but not its order. The complete proof is provided in the corresponding appendix proof section of the revised manuscript.

Thank you for the detailed response. My concerns have been adequately addressed, and I will maintain my positive score.

Reviewer_kb3YOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ProLoRA addresses bidirectional communication costs in federated LoRA fine-tuning. It combines a shared random orthogonal projection compressor with LoRA-aware error feedback, which allows clients to upload compact representations that the server can aggregate and broadcast directly. The error-feedback mechanism filters historical residuals using Gram matrices of the partner LoRA factors, which improves model recovery. The paper provides a non-convex convergence analysis. The evaluation results show substantial communication reductions relative to uncompressed federated LoRA while generally retaining comparable performance.

Critical strengths
  1. The method addresses uplink and downlink compression coherently. The proposed method offers a practical advantage over approaches whose independently selected sparse updates become dense after aggregation. The communication measurements in the evaluation demonstrate this benefit.

  2. The empirical support is substantial and covers relevant trade-offs. The paper reports three-seed results, comparisons under several communication criteria, and ablations of both projection design and error feedback. The appendices extend the evaluation and examine heterogeneity, client count, LoRA rank, runtime, and memory. The reported additional computation is modest in the measured setup. These results strengthen the practical case beyond parameter-count savings alone.

Critical weaknesses
  1. The experiments focus primarily on fully participating, synchronous runs, so the practical communication gains have not been adequately validated outside synchronized deployments. The main experiments use 10 clients, with client scaling ablations restricted to 5, 10, and 20 clients. Although the appendix studies heterogeneity and LoRA rank, it contains no tests for partial participation or client reconnection. This limitation is especially relevant to ProLoRA: clients keep local residuals and assemble global parameters from sequential compressed updates. Reconnecting clients would likely incur extra synchronization and residual-management costs that are not measured here. By comparison, EcoLoRA evaluates 100 total clients with 10 sampled each round, supporting larger pools and asynchronous participation. Small client numbers are reasonable for cross-silo FL in principle, but the current results provide much stronger evidence for this narrow setting than for general cross-device applicability.

  2. Although improvements over vanilla EF are shown, the paper does not include baselines with scalar-damped EF, diagonal residual filters, or ROP run without EF. It is thus unclear whether the gains are driven by LoRA-specific geometric conditioning or by generic residual attenuation. Since the adaptive Gram residual filter is the core methodological contribution, these missing comparisons hinder clean isolation of the proposed mechanism.

Questions for authors
  1. Under the same ROP compressor and communication budget, how does LA-EF compare with scalar-damped EF and ROP without EF? These comparisons would clarify whether the gains require partner-factor geometry rather than generic residual attenuation.

  2. How would a client recover the current global model after missing several rounds, and how would its historical residuals be handled? Can we evaluate the resulting communication costs and model performance under partial participation?

5 comments
AuthorsOpenReview ↗

Author Response to Reviewer kb3Y (Part 1/2)

We thank the reviewer for the careful evaluation of our work and for recognizing ProLoRA's bidirectional communication compression design, communication–accuracy trade-off, and experimental evaluation. To address concerns about partial client participation, stale historical residuals, and the effectiveness of LA-EF's geometry-aware mechanism, we have added method designs, theoretical analyses, and experiments in the revised manuscript. For clarity, W1–W2 refer to Critical Weaknesses, while Q1–Q2 refer to Questions for Authors.

1. Model Synchronization and Reconnection Communication Overhead under Partial Client Participation (W1, Q2)

We thank the reviewer for highlighting model synchronization and reconnection costs under partial participation. We agree that the original ProLoRA primarily targets synchronous full participation, where a client's locally stored global LoRA model may become outdated after multiple missed rounds. We therefore added a Participation-Aware ProLoRA (PA-ProLoRA) extension and a reconnection synchronization mechanism in Appendix E.2.

First, we achieve exact model synchronization using historical compressed aggregate updates. Suppose client ii last synchronized at θvi\boldsymbol{\theta}^{v_i} and rejoins at round tt. The server sends the missed compressed aggregate updates. The client regenerates the orthogonal projection bases from shared random seeds and reconstructs the missing global updates chronologically:

θi←θvi−ηg∑τ=vit−1Qτ(Δˉτ)=θt.\boldsymbol{\theta}_i\leftarrow\boldsymbol{\theta}^{v_i}-\eta_g\sum_{\tau=v_i}^{t-1}\mathcal Q^\tau(\bar{\boldsymbol{\Delta}}^\tau)=\boldsymbol{\theta}^t.

Since ROP's aggregate updates are already compact, synchronization requires no additional server-side compression.

Second, PA-ProLoRA selects the reconnection strategy according to communication cost. When transmitting missed compressed updates costs less than sending the complete LoRA factors, the server uses incremental synchronization. Otherwise, or when historical updates are unavailable, it sends the current complete LoRA factors. Both strategies exactly recover the current global model, and reconnection downlink costs are included in our experiments.

With 100 clients and participation rates of 10% and 30%, PA-ProLoRA incurs 1730 MB and 2089 MB of downlink traffic, compared with 3375 MB and 10125 MB for FedIT. Thus, PA-ProLoRA retains substantial downlink savings even after accounting for reconnection. Full results are in Appendix E.2 and Point 3.

2. Stale Historical Residuals and Participation-Aware Residual Regulation (W1, Q2)

We agree that model synchronization alone does not resolve stale historical residuals. After multiple missed rounds, a client's stored compression residuals may no longer match the geometry of the current global LoRA factors, and directly feeding them back may impair optimization. We therefore added participation-aware historical residual filtering in Appendix E.2.

Inactive clients retain their local residuals without additional feedback. Upon rejoining, they first synchronize the global model, then regulate residuals using the current global partner-factor Gram geometry and participation rate π=k/N\pi=k/N.

Specifically, assuming the interval ℓ\ell between participations follows a geometric distribution, we average the original LA-EF filter over these intervals while fixing the current partner-factor Gram matrix:

Hπ,bt=Eℓ[(Hbt)ℓ]=(Ir+qπSbt)−1.\mathbf H_{\pi,b}^t=\mathbb E_{\ell}[(\mathbf H_b^t)^\ell]=\left(\mathbf I_r+\frac q\pi\mathbf S_b^t\right)^{-1}.

Here, qq is ROP's compression ratio and Sbt\mathbf S_b^t is the normalized Gram matrix of the current global LoRA partner factor. As participation decreases, the filter more strongly attenuates residual directions with greater partner-factor energy, mitigating the effects of stale residuals.

Unlike clearing residuals, this mechanism balances historical information retention and direction-dependent residual suppression. In Appendix E.2, we also give a strictly convex quadratic optimization interpretation and prove that the filtered residual is its unique minimizer.

When π=1\pi=1, the filter reduces to the original LA-EF. Thus, PA-ProLoRA naturally extends the original residual regulation mechanism without changing local LoRA training, ROP's compressed communication representation, or compressed-domain server aggregation, and without additional residual communication. Point 3 compares it experimentally with residual clearing.

AuthorsOpenReview ↗

Author Response to Reviewer kb3Y (Part 2/2)

3. Partial Participation Experiments with 100 Clients and Convergence Guarantees (W1, Q2)

We added 100-client experiments and convergence analysis in Appendix E.2 to validate PA-ProLoRA under partial participation.

We evaluate MNLI over 300 rounds with α=0.5\alpha=0.5, rank r=8r=8, q=1/16q=1/16, and 10% or 30% participation (10 or 30 randomly selected clients per round). Baselines include FedIT, EcoLoRA, ProLoRA, and residual clearing, which discards residuals after ⌈1/π⌉\lceil1/\pi\rceil missed rounds. Communication includes reconnection overhead. Accuracy is mean ± standard deviation over three seeds.

Method10% Accuracy (%)10% Up. (MB)10% Down. (MB)30% Accuracy (%)30% Up. (MB)30% Down. (MB)
FedIT82.42 ± 0.793375337584.14 ± 0.301012510125
EcoLoRA72.05 ± 8.57446297282.58 ± 0.2312698701
ProLoRA68.45 ± 5.51211173077.36 ± 5.726332089
ProLoRA + Clearing78.50 ± 2.57211173080.63 ± 2.446332089
PA-ProLoRA80.76 ± 1.96211173083.09 ± 1.006332089

These results confirm the harm of stale residuals. At 10% and 30% participation, ProLoRA achieves 68.45% and 77.36%, while clearing improves them to 78.50% and 80.63%.

In contrast, PA-ProLoRA reaches 80.76% and 83.09%, improving over original ProLoRA by 12.31% and 5.73%, and over residual clearing by another 2.26% and 2.46%. It also outperforms EcoLoRA by 8.71% and 0.51%, with gaps of just 1.66% and 1.05% from FedIT. Geometric filtering thus recovers more performance than clearing.

Including reconnection synchronization, PA-ProLoRA's total communication costs are 1941 MB and 2722 MB, reducing costs by 71.2% and 86.6% versus FedIT and 43.2% and 72.7% versus EcoLoRA. Thus, reconnection overhead is not excluded.

We also added the complete algorithm (Algorithm 2) and convergence analysis (Theorem 3) in Appendix E.2. Under uniform random client sampling, exact model synchronization, and the stated smoothness, gradient-variance, heterogeneity, and learning-rate conditions, we establish:

1T∑t=1TE[∥∇F(θt)∥2]=O(ΔF+ηg2Kπ(δ2+σ2/E)qηgT+σ2+Eδ2qET).\frac1T\sum_{t=1}^T\mathbb E[\|\nabla F(\boldsymbol{\theta}^t)\|^2]=\mathcal O\left(\frac{\Delta_F+\eta_g^2K_\pi(\delta^2+\sigma^2/E)}{q\eta_g\sqrt T}+\frac{\sigma^2+E\delta^2}{qET}\right).

For fixed positive participation π>0\pi>0 and compression ratio q>0q>0, PA-ProLoRA retains an O(T−1/2)\mathcal O(T^{-1/2}) convergence order. Lower participation may increase the bound's constants without guaranteeing equal convergence speeds. The complete proof is in the appendix.

4. Controlled Ablation Studies on the Benefits of LA-EF's Geometry-Aware Design (W2, Q1)

We agree that the original ablations did not sufficiently separate geometry-aware regulation from damping. We added fixed and tuned scalar damping, diagonal Gram filtering, and ROP without EF in Appendix F.1.

All methods use the same ROP compressor and training conditions: RoBERTa-base/MNLI, 10 clients, 300 rounds, α=0.5\alpha=0.5, and q=1/16q=1/16.

Fixed scalar damping uses HB=HA=(1+q)−1IrH_B=H_A=(1+q)^{-1}I_r. Tuned scalar damping selects the best coefficient from {0.5,0.8,(1+q)−1,0.98}\{0.5,0.8,(1+q)^{-1},0.98\}. Diagonal Gram filtering uses only diagonal entries, while LA-EF uses the full Gram matrix. All ROP variants use the same expanded global learning-rate grid {0.125,0.25,0.5,1,1.5,2}\{0.125,0.25,0.5,1,1.5,2\}, with each independently tuned.

MethodMNLI Accuracy (%)
FedIT (Uncompressed Reference)84.76 ± 0.21
ROP (Without EF)76.80 ± 2.72
ROP + Standard EF73.27 ± 6.52
ROP + Fixed Scalar Damping77.58 ± 6.92
ROP + Tuned Scalar Damping80.35 ± 1.02
ROP + Diagonal Gram Filtering78.54 ± 7.23
ROP + LA-EF (ProLoRA)84.34 ± 0.49

Standard EF can impair ROP, achieving 73.27%, below 76.80% without EF. Fixed and tuned scalar damping improve accuracy to 77.58% and 80.35%, confirming the benefit of reducing feedback strength.

However, LA-EF reaches 84.34%, outperforming the best tuned scalar damping by 3.99%, ROP without EF by 7.54%, and diagonal Gram filtering by 5.80%, while remaining only 0.42% below uncompressed FedIT.

LA-EF's gains cannot be explained by scalar damping alone. Its full partner-factor Gram matrix enables direction-dependent regulation and captures correlations ignored by diagonal filtering, reducing historical residual effects on effective LoRA weights.

Theory appears in Section 4.1 and Appendix B.3, and the settings, unified learning-rate search, and ablations in Appendix F.1.

Thank you for the update. The new experimental results have resolved my concern. I will keep my positive rating.

Reviewer_K7M4OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies communication-efficient federated fine-tuning with LoRA. It argues that existing approaches either restrict the trainable LoRA factors (FFA-LoRA, RoLoRA, Fed-SB), which hurts accuracy, or sparsify the updates (EcoLoRA, FedSRD). Sparsified updates from different clients become dense after aggregation, so the downlink stays expensive or the server must recompress with extra error.

The authors propose ProLoRA, which has two components. (1) A random orthogonal projection compressor applies a shared-seed, per-round orthonormal basis to the LoRA factors B and A separately. Because the projection is linear, the server averages the compressed matrices directly and broadcasts the result at the same size. Clients reconstruct the update with the same basis, so uplink and downlink both cost about q·d. (2) LoRA-aware error feedback stores each client’s compression residual and adds it back in the next round through a filter. Besides the method, the paper also proves an O(1/√T) convergence rate, matching full-precision FedAvg.

Critical strengths
  1. A well-targeted design with a strong communication/accuracy trade-off. The paper correctly identifies that sparsification-based compressors densify after aggregation, which forces either a dense downlink or server-side recompression. It builds a compressor whose linearity lets the server average compressed LoRA factors directly, giving symmetric ~q·d uplink and downlink while still training both A and B. This matters because it delivers the practical promise: about 94% less communication at q=1/16 with accuracy close to FedIT on RoBERTa-base (+0.11 average over five GLUE tasks), with a similar picture on Llama-2-7B and Gemma-7B (App. E.2). Rank-restriction methods are less stable (e.g., Fed-SB gets 54.6±10.2 on MNLI).

  2. A careful evaluation protocol with component ablations. Methods are compared under equal rounds, equal communication budget and equal target accuracy, with three seeds. The ablations support the design choices: matrix-form ROP is 8.5× faster than vector projection with lower peak memory (54.03→33.06 MB), and LA-EF improves on standard EF for q∈{1/8, 1/16, 1/32}. Overhead relative to FedIT is small (about 1.6% round time on RoBERTa-base, 5.9% on Llama-2-7B). This makes the empirical claims easier to trust than a single headline table would.

Critical weaknesses
  1. Limited novelty, and the closest prior work is neither cited nor compared. The central idea is a shared-seed linear compressor, so that the server can aggregate in the compressed domain. It is not new. FetchSGD (Rothchild et al., 2020) aggregates linear Count Sketches at the server with error accumulation. FedKSeed (ICML 2024) and FedKRSO (2026) use shared seeds to regenerate random directions or subspaces, and the server averages per-seed quantities. PowerSGD and EF21 are standard references for low-rank compression and error feedback. The paper cites classic EF but none of these. What is specific to this paper is a random orthonormal projection applied to each LoRA factor in matrix form, which is largely an efficiency design, and the LA-EF filter, a modest LoRA-specific modification of standard EF. All baselines are restriction- or sparsification-based, and no linear-sketching or random-subspace method is included. FedKRSO in particular is evaluated on the same setup (GLUE, RoBERTa-base, 10 clients, Dirichlet splits) against FedIT and FFA-LoRA, so a direct comparison would have been straightforward. As it stands, the experiments cannot show that the gains come from the new components rather than from using a linear compressor at all. This bears directly on the paper’s significance.

  2. The “no loss of accuracy” claim holds only under mild heterogeneity, and the headline gains are framed against weak comparators. At the default α=0.5, accuracy is on par with FedIT (+0.11 on average in Table 2; Gemma-7B MATH is 10.78 vs 11.41), so the contribution is communication savings, not accuracy. Under stronger heterogeneity this breaks down: in Appendix Table 6 at α=0.3, MNLI drops from 83.74±0.12 (FedIT) to 79.84±2.82 (ProLoRA), a gap of about 3.9 points, whereas the text only reports the 0.52-point QNLI gap. The “up to 10.14%” improvement in the equal-budget setting (Table 3) is measured against the weakest baseline (Fed-SB). Against the strongest baseline (RoLoRA) the average gain is 3.84 points, and the budget (FedIT’s cost for 20 rounds) favors methods that are cheap per round. The paper motivates communication as the bottleneck, yet it reports no wall-clock or bandwidth-constrained evaluation. It also has no experiments with partial client participation, although EF keeps per-client residual state that becomes stale when clients are sampled. Together these leave unclear in which realistic settings the method keeps its advantage.

Questions for authors
  1. Related linear-compression and random-subspace methods. FetchSGD (Rothchild et al., 2020), FedKSeed (Qin et al., 2024) and FedKRSO share the core ingredients of ProLoRA: shared-seed linear compression with aggregation in the compressed domain. FedKRSO is evaluated on the same setup (GLUE, RoBERTa-base, 10 clients, Dirichlet splits) against FedIT and FFA-LoRA. Could the authors compare against at least one of these under a matched per-round communication budget, or state precisely what ProLoRA achieves that they cannot? If the answer is that these are full-parameter methods, a matched-budget comparison would still show whether training both LoRA factors with error feedback is better than training within a random subspace.

  2. Partial participation. Algorithm 1 and the analysis behind Theorem 1 (Assumptions 1–3 and the client averaging in App. D.3) assume that all N clients participate in every round. LA-EF keeps a per-client residual and re-injects it using filters computed from the current global factors, so under client sampling the residuals become stale. The paper itself cites Li & Li (2023), which studies error feedback under partial participation, but does not address this setting. How does ProLoRA perform when only a fraction of clients participate per round (e.g., 10–50%), and does the convergence guarantee extend to partial participation?

2 comments
AuthorsOpenReview ↗

Author Response to Reviewer K7M4 (Part 1/2)

We thank the reviewer for recognizing ProLoRA's bidirectional communication design and communication–accuracy trade-off. For clarity, W1–W2 refer to Critical Weaknesses, and Q1–Q2 to Questions for Authors.

1. Differences from Existing Linear-Compression and Random-Subspace Methods and Direct Comparisons (W1, Q1)

ProLoRA's novelty lies in the joint design of ROP and LA-EF, not in linear compression alone. FetchSGD uses linear Count Sketch aggregation but reconstructs and sparsifies TopK updates for downlink. FedKSeed combines shared seeds with zeroth-order optimization, FedKRSO uses shared random subspaces, PowerSGD compresses gradients through low-rank approximation, and EF21 studies error feedback.

ROP instead applies matrix-form orthogonal projection to LoRA's two-factor structure. It allows direct aggregation and broadcast of compressed updates without server-side recompression, while both factors remain fully trainable locally. We further identify the historical-feedback problem arising when ROP is combined with standard EF, and introduce geometry-aware LA-EF for model recovery. We expanded related work in Section 2 and added the theoretical rationale in Section 4.1 and Appendix B.

We also added direct FetchSGD and FedKRSO comparisons in Appendix F.7. Experiments use RoBERTa-base on MNLI/QNLI, 10 clients, 300 rounds, and Dirichlet α=0.5\alpha=0.5. FedKRSO is restricted to Query/Value with K=I=rKRSO=1K=I=r_{\mathrm{KRSO}}=1, matching ProLoRA's uplink and downlink budgets. FetchSGD matches the uplink budget, with its full downlink cost included.

MethodUplink (MB)Downlink (MB)Total (MB)MNLI (%)QNLI (%)
FedIT33753375675084.76 ± 0.2191.61 ± 0.24
FetchSGD21131652777.03 ± 2.3187.36 ± 0.47
FedKRSO21121142281.40 ± 0.4189.21 ± 0.34
ProLoRA21121142284.34 ± 0.4991.35 ± 0.59

At identical bidirectional communication costs, ProLoRA improves over FedKRSO by 2.94%/2.14% on MNLI/QNLI. Against FetchSGD at matched uplink, it improves accuracy by 7.31%/3.99% with less total communication. Relative to uncompressed FedIT, ProLoRA achieves comparable accuracy with 93.75% less total communication. These results support the combined benefits of ROP and LA-EF.

2. Scope of Performance Preservation and Comparisons under Different Communication Budgets (W2)

We do not claim lossless accuracy under every level of data heterogeneity. At α=0.3\alpha=0.3, ProLoRA achieves 79.84 ± 2.82% MNLI accuracy versus FedIT's 83.74 ± 0.12%, a 3.90% gap. On QNLI the gap is 0.52% (90.62% versus 91.14%). Nevertheless, ProLoRA outperforms the evaluated communication-efficient baselines with 93.75% lower total communication than FedIT. We clarified this limitation in Appendix F.4 and qualified our performance claims.

We also clarified the original 450 MB equal-budget comparison (20 FedIT rounds) in Section 6. Across five GLUE tasks, ProLoRA averages 88.93%, exceeding the strongest competing baseline, RoLoRA (85.09%), by 3.84%, and the weakest, Fed-SB (78.79%), by 10.14%. We now explicitly report the 3.84%–10.14% range rather than implying the larger gain is against the strongest baseline.

To test other budgets, Appendix F.8 adds 225 MB and 900 MB experiments, equivalent to 10 and 40 FedIT rounds. We report the last evaluation within each budget using existing trajectories (at most 300 rounds):

MethodMNLI 225 MBMNLI 900 MBQNLI 225 MBQNLI 900 MBQQP 225 MBQQP 900 MB
FedIT68.56 ± 3.5081.03 ± 0.2683.43 ± 0.7388.58 ± 0.4477.65 ± 0.9683.81 ± 0.34
FFA-LoRA65.76 ± 3.4978.10 ± 0.9881.14 ± 3.6686.80 ± 0.7978.65 ± 0.7282.34 ± 0.67
RoLoRA62.68 ± 1.2481.07 ± 0.1783.10 ± 1.6988.44 ± 0.6480.56 ± 0.9384.04 ± 0.31
Fed-SB53.54 ± 9.7054.57 ± 10.2185.14 ± 1.0485.70 ± 0.8779.77 ± 1.5780.59 ± 0.90
EcoLoRA41.78 ± 16.3777.64 ± 4.8880.07 ± 2.4289.80 ± 0.6276.71 ± 1.9084.13 ± 1.43
FedSRD72.17 ± 3.3278.35 ± 1.3584.81 ± 0.7487.32 ± 0.7679.49 ± 0.4982.76 ± 0.43
ProLoRA82.90 ± 1.0784.35 ± 0.5090.46 ± 0.7791.31 ± 0.6285.99 ± 0.5586.81 ± 0.31

ProLoRA leads in all six task–budget combinations. Its gains over each task's strongest competitor are 5.32%–10.73% at 225 MB and 1.51%–3.28% at 900 MB. Thus, its communication–accuracy advantages are not confined to the original 450 MB budget.

AuthorsOpenReview ↗

Author Response to Reviewer K7M4 (Part 2/2)

3. End-to-End Training Efficiency under Bandwidth Constraints (W2)

Appendix F.3 adds training-time evaluations to 75% MNLI accuracy using 10 parallel clients under symmetric (0.5/0.5, 1/1, 2/2 Mbps) and asymmetric (0.5/1, 1/2, 2/4 Mbps) uplink/downlink bandwidths. We combine measured computation time with bandwidth-controlled TCP transmission time at RTT = 40 ms.

Method0.5/0.51/12/20.5/11/22/4
FedIT13.26 ± 0.006.97 ± 0.003.82 ± 0.0010.12 ± 0.005.40 ± 0.003.04 ± 0.00
FFA-LoRA17.43 ± 3.499.57 ± 1.915.64 ± 1.1313.50 ± 2.707.60 ± 1.524.65 ± 0.93
RoLoRA12.79 ± 2.017.02 ± 1.114.14 ± 0.659.91 ± 1.565.58 ± 0.883.42 ± 0.54
Fed-SBNRNRNRNRNRNR
EcoLoRA18.07 ± 4.429.94 ± 2.445.88 ± 1.4411.62 ± 2.836.72 ± 1.644.27 ± 1.04
FedSRD19.82 ± 7.5415.06 ± 5.7412.67 ± 4.8417.04 ± 6.4813.66 ± 5.2111.98 ± 4.57
ProLoRA5.38 ± 2.034.00 ± 1.513.31 ± 1.254.69 ± 1.773.66 ± 1.383.14 ± 1.19

Time: minutes. Bandwidth: Mbps (uplink/downlink). NR: target accuracy not reached.

ProLoRA is fastest in five of six settings. Against FedIT, it reduces time by 59.4% at 0.5/0.5 Mbps (13.26 to 5.38 minutes), 42.6% at 1/1 Mbps, and 53.7% at 0.5/1 Mbps. Thus, compression improves training efficiency under constrained bandwidths.

4. Partial Client Participation, Reconnection Synchronization, and Convergence Guarantees (W2, Q2)

Appendix E.2 adds Participation-Aware ProLoRA (PA-ProLoRA) with reconnection synchronization, residual filtering, Algorithm 2, Theorem 3, and experiments.

Reconnection synchronization. Returning clients replay missing compressed aggregate updates using projection bases regenerated from shared seeds. If replay is costlier or unavailable, the server sends full global LoRA factors. Both ensure exact synchronization, with reconnection downlink counted.

Stale residual regulation. Inactive clients retain their residuals. On rejoining, the filter uses participation rate π=k/N\pi=k/N and current partner-factor geometry. Averaging over geometric participation intervals with a fixed current Gram matrix gives

Hπ,bt=Eℓ[(Hbt)ℓ]=(Ir+qπSbt)−1.\mathbf H_{\pi,b}^t=\mathbb E_{\ell}[(\mathbf H_b^t)^\ell]=\left(\mathbf I_r+\frac q\pi\mathbf S_b^t\right)^{-1}.

At π=1\pi=1, this reduces to LA-EF. Lower participation strengthens attenuation along directions of high partner-factor energy, mitigating stale residuals without discarding all historical information. The server's compressed-domain aggregation is unchanged.

We evaluated MNLI with 100 clients, 300 rounds, α=0.5\alpha=0.5, r=8r=8, q=1/16q=1/16, and 10%/30% participation. Residual clearing drops residuals after ⌈1/π⌉\lceil1/\pi\rceil missed rounds. All communication includes reconnection.

Method10% Accuracy (%)10% Up. (MB)10% Down. (MB)30% Accuracy (%)30% Up. (MB)30% Down. (MB)
FedIT82.42 ± 0.793375337584.14 ± 0.301012510125
EcoLoRA72.05 ± 8.57446297282.58 ± 0.2312698701
ProLoRA68.45 ± 5.51211173077.36 ± 5.726332089
ProLoRA + Clearing78.50 ± 2.57211173080.63 ± 2.446332089
PA-ProLoRA80.76 ± 1.96211173083.09 ± 1.006332089

The original ProLoRA drops to 68.45%/77.36%, while residual clearing improves it to 78.50%/80.63%. PA-ProLoRA reaches 80.76%/83.09% accuracy, improving on original ProLoRA by 12.31%/5.73% and residual clearing by 2.26%/2.46% at 10%/30% participation. Compared with EcoLoRA, the gains are 8.71%/0.51%. This demonstrates that filtering stale residuals is more effective than simply discarding them. Including reconnection, total communication is 1941/2722 MB, saving 71.2%/86.6% versus FedIT and 43.2%/72.7% versus EcoLoRA.

Theorem 3 (Appendix E.2) establishes, under uniform sampling, exact synchronization, and the stated smoothness, variance, heterogeneity, and learning-rate assumptions:

1T∑t=1TE[∥∇F(θt)∥2]=O(ΔF+ηg2Kπ(δ2+σ2/E)qηgT+σ2+Eδ2qET).\frac1T\sum_{t=1}^T\mathbb E[\|\nabla F(\boldsymbol{\theta}^t)\|^2]=\mathcal O\left(\frac{\Delta_F+\eta_g^2K_\pi(\delta^2+\sigma^2/E)}{q\eta_g\sqrt T}+\frac{\sigma^2+E\delta^2}{qET}\right).

For fixed π,q>0\pi,q>0, PA-ProLoRA retains an O(T−1/2)\mathcal O(T^{-1/2}) convergence order. A lower participation rate increases the bound's constants, so equal convergence speeds are not guaranteed. The full proof is in Appendix D.5.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of Revisions and Invitation for Further Discussion

We sincerely thank all reviewers for their careful evaluations and constructive feedback.

During the discussion period, we have provided detailed point-by-point responses and uploaded a revised manuscript. To facilitate further discussion, we summarize the major revisions and their corresponding reviewer concerns below.

1. Methodological Novelty and Comparisons with Existing Methods (7HJb: W1, K7M4: W1/Q1) (Section 2, Appendix F.7).

2. Theoretical Justification of LA-EF and Geometry-Aware Residual Regulation (7HJb: W1/Q1/Q2, grNC: W1/Q1, kb3Y: W2/Q1) (Section 4.1, Appendices B.1–B.3 and F.1).

3. Fairness of Global Learning-Rate Tuning (grNC: Q2) (Section 6.4, Appendix F.1).

4. Performance Limitations and Communication Efficiency Comparisons (K7M4: W2, grNC: W2/Q3) (Section 6, Appendices F.4 and F.8).

5. Training Efficiency under Bandwidth Constraints (K7M4: W2) (Appendix F.3).

6. Partial Client Participation and Convergence Analysis (K7M4: W2/Q2, kb3Y: W1/Q2) (Appendix E.2, Algorithm 2, Theorem 3).

7. Two-Factor Aggregation Bias (7HJb: W2, addressed in our response) and Rectangular LoRA Layers (grNC: Q4, Appendix E.1, Theorem 2).

We hope that these additional analyses, experiments, and clarifications adequately address the reviewers' concerns.

With only about three days remaining before the author–reviewer discussion closes on November 18, 2026 (AoE), we sincerely encourage all reviewers to participate in the discussion and share their feedback on our responses and revised manuscript. In particular, we would greatly appreciate knowing whether our revisions have addressed your concerns or if any issues remain unresolved.

Your timely feedback would give us a valuable opportunity to provide further clarification before the discussion closes. We remain fully available to address any additional questions.

Thank you again for your time and consideration.

AuthorsOpenReview ↗

Response to Reviewer K7M4's Follow-up Comments and Additional Revisions

Thank you for reading the revision and recognizing the new comparisons and ablations. We appreciate your increased score and feedback.

1. Contributions and Baseline Fairness (W1/Q1)

We agree that LA-EF is our main methodological contribution. We also consider ROP's efficient bidirectional linear compression design and identifying its optimization issue with standard EF important contributions. We clarify this in the Introduction.

Appendix F.7 specifies a common local learning-rate grid {10−3,5×10−4,10−4,5×10−5,2×10−5}\lbrace 10^{-3},5\times10^{-4},10^{-4},5\times10^{-5},2\times10^{-5}\rbrace and global grid {1,1.5,2}\lbrace 1,1.5,2\rbrace for separately tuning FedKRSO, FetchSGD, and ProLoRA. With K=I=rKRSO=1K=I=r_{\mathrm{KRSO}}=1 and J=20J=20, FedKRSO performs IJ=20IJ=20 local update steps per round, matching the other two methods.

2. Strong Heterogeneity (W2(a))

We qualified the abstract, Introduction, and Conclusion. The main text now reports the MNLI gap at α=0.3\alpha=0.3: 79.84% (ProLoRA) versus 83.74% (FedIT), a 3.90-point gap. We acknowledge this limitation under stronger heterogeneity.

3. Communication Budgets (W2(b))

Appendix F.8 clarifies that budgets are upper limits, with a 300-round cap. ProLoRA completes 300 rounds with 422 MB, so its 900 MB and 450 MB results are identical, whereas FedIT runs only 40 rounds at 900 MB.

4. Rounds-to-Target and Training Time (W2(c))

Appendix F.3 reports rounds-to-target and training times at 75% and 82% MNLI accuracy (three seeds, mean ± SD). NR means at least one seed failed to reach the target within 300 rounds.

At 75%, ProLoRA takes 70.00 rounds versus FedIT's 20.00, yet is fastest in five of six bandwidth settings. It saves 59.4% time at 0.5/0.5 Mbps, while the times are close at 2/4 Mbps (3.14 versus 3.04 min). At 82%, it takes 126.67 versus 53.33 rounds but is fastest in all six settings. It saves 72.5% at 0.5/0.5 Mbps and 29.8% even at 2/4 Mbps (5.69 versus 8.10 min).

75% target (minutes, Mbps up/down):

MethodRounds0.5/0.51/12/20.5/11/22/4
FedIT20.00±0.0013.26±0.006.97±0.003.82±0.0010.12±0.005.40±0.003.04±0.00
FFA-LoRA50.00±10.0017.43±3.499.57±1.915.64±1.1313.50±2.707.60±1.524.65±0.93
RoLoRA36.67±5.7712.79±2.017.02±1.114.14±0.659.91±1.565.58±0.883.42±0.54
Fed-SBNRNRNRNRNRNRNR
EcoLoRA46.67±11.5518.07±4.429.94±2.445.88±1.4411.62±2.836.72±1.644.27±1.04
FedSRD153.33±58.5919.82±7.5415.06±5.7412.67±4.8417.04±6.4813.66±5.2111.98±4.57
ProLoRA70.00±26.465.38±2.034.00±1.513.31±1.254.69±1.773.66±1.383.14±1.19

82% target (same units):

MethodRounds0.5/0.51/12/20.5/11/22/4
FedIT53.33±5.7735.36±3.8318.58±2.0110.20±1.1026.98±2.9214.39±1.568.10±0.88
FFA-LoRANRNRNRNRNRNRNR
RoLoRA120.00±0.0041.87±0.0022.99±0.0013.55±0.0032.43±0.0018.27±0.0011.19±0.00
Fed-SBNRNRNRNRNRNRNR
EcoLoRA225.67±126.1584.37±46.7146.57±25.8127.69±15.3753.97±29.8431.38±17.3720.08±11.14
FedSRDNRNRNRNRNRNRNR
ProLoRA126.67±37.869.73±2.917.24±2.166.00±1.798.49±2.546.62±1.985.69±1.70

Lower per-round communication offsets the extra rounds, including at 82%.

5. Partial Participation (W2(d))

We agree Table 1's symmetric costs assume full participation. Returning clients incur extra downlink synchronization. The System Model and Experiments state this scope, and measured costs include synchronization. At 10% participation, PA-ProLoRA improves accuracy from 68.45% to 80.76%, 1.66 points below FedIT (82.42%), while reducing total communication by 71.2%.

We strengthened Theorem 3 using ηg=Θ(Ek)\eta_g=\Theta(\sqrt{Ek}) and a refined separation of client-sampling noise and local drift. Its new bound is:

1T∑t=1TE[∥∇F(θt)∥2]=Oq,π(ΔF+σπ2TEk+σ2+Eδ2ET+σπ2+ρGK2T).\frac1T\sum_{t=1}^T\mathbb E[\|\nabla F(\mathbf{\theta}^t)\|^2]=\mathcal O_{q,\pi}\left(\frac{\Delta_F+\sigma_\pi^2}{\sqrt{TEk}}+\frac{\sigma^2+E\delta^2}{ET}+\frac{\sigma_\pi^2+\rho G_{\mathcal K}^2}{T}\right).

Here π=k/N\pi=k/N, ρ=(N−k)/(k(N−1))\rho=(N-k)/(k(N-1)), χ=kρ=(N−k)/(N−1)\chi=k\rho=(N-k)/(N-1), and σπ2=σ2+Eχδ2\sigma_\pi^2=\sigma^2+E\chi\delta^2. For fixed positive q,πq,\pi, the leading O(1/TEk)\mathcal O(1/\sqrt{TEk}) term explicitly captures speedup with participating clients kk. Client sampling contributes Eχδ2E\chi\delta^2 at leading order, whereas local drift contributes δ2/T\delta^2/T. Sampling heterogeneity remains for k<Nk<N. When k=Nk=N, client-sampling noise vanishes and the proof recovers Theorem 1. The full proof is in the appendix.

6. Table Discrepancies (W2(e))

FedIT and ProLoRA in the heterogeneity, client-count, and rank studies were independently rerun using three seeds different from Table 2. This explains the slight differences. Each table's discussion now states this.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.