acceptodds
Withdrawn from ICLR 2027

FedLore: Communication and Memory Efficient Federated Learning via Shared Gradient Low-Rank Projection

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Federated training of foundation models is constrained by client memory and communication costs. LoRA-based methods reduce these costs through low-rank adapters, but their fixed rank budget can limit adaptation. Gradient low-rank optimization offers greater flexibility, yet independently chosen client subspaces create a problem we term subspace fragmentation: local projections interact with data heterogeneity to bias aggregated directions, while aggregation can increase update rank and communication cost. Thus, accurate local gradient compression need not preserve global descent. We propose FedLore, which shares a low-rank optimization basis within each round and refreshes it across rounds. The shared basis enables exact aggregation in low-rank coordinates and eliminates the identified projection bias. Subspace refresh allows the accumulated model update to exceed the per-round rank budget. We characterize the aggregation bias and establish an stationarity bound for the projected-SGD variant under a global-gradient coverage condition and standard smoothness and variance assumptions, with bounded gradient heterogeneity. Experiments on vision and language tasks, including federated pre-training, show that FedLore outperforms the evaluated low-rank adapter baselines and matches or exceeds full-parameter training, while reducing communication and optimizer-state memory.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Federated Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_xpiQOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes FedLore, a shared low-rank projection method for communication- and memory-efficient federated foundation model training, supported by theoretical analysis and experiments on vision and language tasks.

Critical strengths
  1. FedLore addresses communication and memory bottlenecks in federated foundation model training through shared low-rank projections and dynamic subspace updates. The method is practically motivated. The paper also analyzes the aggregation bias arising from projection disagreement and data heterogeneity, providing theoretical support through bias bounds and conditional convergence guarantees for projected SGD.
  2. FedLore was evaluated on a range of vision and language tasks, including LLaMA pre-training at different model scales. The broad experimental evaluation demonstrates competitive performance of the proposed method.
Critical weaknesses
  1. FedSub [1] and FedSLoP [2] have already explored shared and dynamically updated low-dimensional subspaces in federated learning. Although FedLore differs in subspace selection and communication mechanisms, the lack of discussion and direct comparisons makes its additional contributions unclear.
  2. The nonconvex convergence guarantee relies on the Aggregate global-gradient coverage assumption (Assumption 4), which is not guaranteed by the implemented subspace selection strategy. Moreover, the analysis focuses on projected SGD, whereas the experiments use AdamW, limiting the theoretical support for the actual algorithm.

References

[1] Zhang, J., Fan, Y., Xu, Y., & Yuan, K. (2026). An efficient subspace algorithm for federated learning on heterogeneous data. IEEE Transactions on Signal Processing.

[2] He, Y., Huang, Z., Geng, J., & Yuan, K. (2026). FedSLoP: Memory-Efficient Federated Learning with Low-Rank Gradient Projection. arXiv preprint arXiv:2604.24012.

Questions for authors
  1. What are FedLore's key contributions relative to FedSub and FedSLoP, particularly in subspace construction and model synchronization?
  2. Can the nonconvex convergence guarantee in Theorem 2 (established for projected SGD) be extended to AdamW, which is used in the actual implementation?
  3. The experiments only consider full client participation, whereas partial participation is more common in practical federated learning. How does partial participation affect FedLore's model performance, synchronization, and communication efficiency?
Reviewer_XvEEOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

FedLore shares a low-rank gradient projection across clients in each round and updates the projection basis across rounds. The paper explains how client-specific projections can interact with data heterogeneity and analyzes convergence for projected SGD. Experiments evaluate accuracy, communication, and optimizer-state memory on vision and language tasks, including language-model pretraining.

Critical strengths

• The paper identifies an aggregation problem caused by clients using different gradient projections. This gives a concrete reason to share the projection basis within each round.

• The evaluation includes vision tasks, language fine-tuning, and language-model pretraining. Across the reported settings, FedLore reduces communication and optimizer-state memory while retaining competitive performance.

Critical weaknesses

• The cancellation result is established for projected SGD, but the experiments use AdamW. AdamW rescales updates using client-specific optimizer states. Does the same cancellation still hold under these updates? The paper needs to clarify whether its theoretical explanation applies to the experimental optimizer.

• The pretraining experiments include few methods that update their low-rank subspaces during training or accumulate updates beyond a fixed rank. These methods should be included to test whether FedLore provides further improvements.

• Can the communication savings carry over to partial participation? The experiments use full participation, and low-rank synchronization depends on current model caches. Please explain how a returning client would recover missed updates and how much communication this would require.

Questions for authors

Please see weaknesses.

Reviewer_5M7uOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies Federated Learning of large models with low rank gradient projection. It target a known bias in FedLora setting, where client low ranks updates become biased under data heterogeneity when they are left to chose their own projection spaces. Instead, they propose to share the projection basis across clients for each round and refreshing it after every round, resulting in exact aggregation and keep true low rank. They present convergence proof as well as empirical results on large models.

Critical strengths
  1. Present clear motivation of need for shared basis, alongwith evidence and a counterexample where despite the projection being accurate, the training stalls.

  2. The method is intuitive and simple. The paper is easy to follow and nicely written.

Critical weaknesses

W1. The training setup is very small. From the reported settings of 20 clients with 100 rounds * 50 local steps, the training amounts to only 0.4B tokens which is very small for such large models going upto LLaMA-1B. Comparatively, the similar techniques like GaLore trained their models in vastly bigger settings.

W2. Why does FedLore-Random using random projection beat the full training in Table 3. This further suggests that it is the short-training effect and not due to a better optimizer.

W3. Limited baselines are evaluated. The only low-rank adapter baseline is FedIT, which itself cannot pre-train from scratch. As in table. 3, its PPL is in thousands. Other similar techniques are mentioned in the paper e.g., FRLoRA, but no comparison is provided.

W4. Although, convergence is stated for partial participation, but experimental results are missing from the paper.

W5. A similar paper PowerSGD[1], uses similar core idea for sharing projection across workers, but it is not cited. Also, the novelty of the core idea seems marginal.

[1] Vogels, Thijs, Sai Praneeth Karimireddy, and Martin Jaggi. "PowerSGD: Practical low-rank gradient compression for distributed optimization." Advances in Neural Information Processing Systems 32 (2019).

Questions for authors

Q1. The convergence result assumes that the share projection covers the global gradient, can you show the FedLore's projection does too?

Q2. Does the results hold for longer training budgets or with different learning rates?

Q3. Why does FedLore-Random beat FedFull?

Q4. Did you measure the peak client memory and time per step? Or is it derived?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.