- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper proposes ProLoRA, a communication-efficient federated LoRA fine-tuning framework combining Random Orthogonal Projection and LoRA-Aware Error Feedback. ROP enables direct aggregation and broadcasting of compressed LoRA updates, while LA-EF exploits the coupling between LoRA factors to regulate historical compression residuals. The authors provide convergence analysis and evaluate the method across multiple datasets and language models, demonstrating substantial communication reduction while maintaining competitive performance.
Critical strengths
The paper addresses bidirectional communication overhead in federated LoRA fine-tuning, an important challenge when communication resources are limited. The shared orthogonal projection design allows for the compression of update parameters, thereby reducing communication costs.
Critical weaknesses
- Limited methodological novelty and insufficient justification of LA-EF. The core ROP design builds on well-established random projection and linear aggregation properties, while error feedback is also a widely studied technique. Thus, the main methodological novelty appears to lie in LA-EF. The article also lacks clarity regarding the design rationale and details of the LA-EF.
- Aggregation bias still exists, which affects federated training performance. When ProLoRA is aggregated in the cloud, it still behaves similarly to averaging over the same A or B factors. Due to the bilinear nature of LoRA, this can lead to aggregation bias issues. LA-EF primarily compensates for compression errors, rather than factor aggregation errors.
Questions for authors
- Under the same compression ratio and optimization budget, what advantages does LA-EF offer over simpler residual damping or adaptive filtering strategies?
- LA-EF attenuates historical compression residuals using the Gram matrices of LoRA factors, potentially suppressing useful accumulated information. Can the authors provide stronger theoretical justification for this trade-off between noise reduction and information preservation?
5 comments
Author Response to Reviewer 7HJb (Part 1/2)
We thank the reviewer for the careful evaluation of our work and for recognizing the value of bidirectional communication compression. For clarity, W1–W2 refer to Critical Weaknesses, while Q1–Q2 refer to Questions for Authors.
1. Methodological Novelty and Design Motivation of ROP and LA-EF (W1)
We believe that the novelty of ProLoRA lies in the joint design of ROP and LA-EF, rather than a simple combination of existing random projection and error feedback techniques. ROP constructs a compact bidirectional communication representation tailored to the two-factor structure of LoRA, enabling the server to directly aggregate and broadcast compressed updates without additional compression while keeping both LoRA factors fully trainable locally. We have expanded the related-work discussion in Section 2 and added direct comparisons with FetchSGD and FedKRSO in Appendix F.7 of the revised manuscript.
More importantly, we find that directly combining ROP with standard EF may impair model optimization. To explain this phenomenon, we have added a stochastic quadratic model analysis in Section 4.1 and Appendix B.1, showing that when both methods are mean-square stable and persistent perturbation is nonzero, standard EF introduces an additional historical-feedback penalty:
We further reveal in Appendix B.2 that the Gram spectrum of the partner LoRA factor makes historical-feedback costs direction-dependent. Motivated by this finding, LA-EF regulates historical residuals using the global LoRA geometry while preserving ROP's compact communication representation without additional communication overhead. Its quadratic optimization objective and closed-form solution have been added to Section 4.1 and Appendix B.3.
2. Theoretical Justification of LA-EF and the Trade-off Between Residual Retention and Noise Suppression (W1, Q2)
LA-EF aims to balance historical information retention and projection noise suppression, rather than simply reducing historical residuals. We have added the following theoretical analyses in Section 4.1 and Appendix B.3 of the revised manuscript.
For historical residuals , when the two factors use independent random orthogonal projections with the same retention ratio, the first-order projection noise in the effective LoRA weight space satisfies
This result reveals the dependence of projection noise on the geometry of the partner factors. Minimizing this noise alone yields the trivial zero-residual solution. Therefore, we formulate a strictly convex quadratic objective that jointly controls historical information loss and direction-dependent projection noise:
Here, . For nonzero Gram matrices, this objective admits a unique closed-form solution:
For zero Gram matrices, we use an isotropic filter as a fallback.
Furthermore, we establish a direction-dependent theoretical advantage of LA-EF over scalar damping under the same residual norm (Appendix B.3):
Here, satisfies . This result shows that, under the same residual norm, LA-EF produces no greater partner-weighted residual effect than uniform scalar damping, demonstrating the theoretical value of direction-dependent filtering.
Dear Reviewer 7HJb,
Thank you for your valuable feedback on our paper. As only one day remains in the discussion period, we would greatly appreciate it if you could review our rebuttal and revised manuscript and share any remaining questions or concerns. We would be happy to provide further clarification and engage in discussion.
Over the past few days, we have engaged in extensive discussions with other reviewers, particularly Reviewer K7M4. In addition to our rebuttal and revised manuscript, we sincerely invite you to review these discussions, which provide further clarifications on several important aspects of our work. We believe these clarifications and discussions may help provide a more comprehensive understanding of our contributions. We would be very grateful if you could reconsider your assessment in light of these updates.
Thank you again for your time and consideration.
Author Response to Reviewer 7HJb (Part 2/2)
3. Advantages of LA-EF over Simple Residual Damping and Alternative Filtering Strategies (W1, Q1)
We have added controlled ablation experiments in Appendix F.1 of the revised manuscript. All methods use the same ROP compressor and are evaluated on MNLI with 10 clients, 300 communication rounds, α=0.5, and q=1/16. Fixed scalar damping uses c=(1+q)−1, while tuned scalar damping selects the best coefficient from {0.5,0.8,(1+q)−1,0.98}. Diagonal Gram filtering uses only the diagonal entries of the Gram matrix. All ROP variants use the same expanded global learning-rate search grid {0.125,0.25,0.5,1,1.5,2}.
Simple scalar damping can indeed mitigate the negative effects of standard EF, but it cannot match LA-EF. LA-EF outperforms the best tuned scalar damping strategy by 3.99% and diagonal Gram filtering by 5.80%, while remaining only 0.42% below uncompressed FedIT. These results demonstrate that LA-EF's performance gains do not arise solely from uniformly attenuating historical residuals, and that the complete LoRA geometry provides additional benefits.
4. Aggregation Bias Arising from the Two-Factor Structure of LoRA (W2)
We agree that separately aggregating the two LoRA factors is generally not equivalent to directly aggregating the effective weights. Let Bˉ=∑ipiBi and Aˉ=∑ipiAi. Then,
∑ipiBiAi−BˉAˉ=∑ipi(Bi−Bˉ)(Ai−Aˉ).
However, the two-factor aggregation bias and ROP's communication compression error are distinct problems. The former may exist even without communication compression, while LA-EF addresses only the latter without modifying the server-side aggregation rule.
To evaluate its practical impact, we conducted an additional comparison between FedIT and FedEx-LoRA (Singhal et al., ACL 2025). FedEx-LoRA achieves exact aggregation of effective weight updates by introducing a correction residual. We evaluate both methods on MNLI with 10 clients, 300 communication rounds, and α=0.5.
Under this setting, exact aggregation does not yield a noticeable accuracy improvement. This does not imply that aggregation bias is unimportant in other settings. ProLoRA focuses on bidirectional communication compression and historical residual feedback rather than eliminating two-factor aggregation bias.
Singhal, Ponkshe, and Vepakomma. “FedEx-LoRA: Exact Aggregation for Federated and Efficient Fine-Tuning of Large Language Models.” ACL 2025.