- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes FedLore, a shared low-rank projection method for communication- and memory-efficient federated foundation model training, supported by theoretical analysis and experiments on vision and language tasks.
Critical strengths
- FedLore addresses communication and memory bottlenecks in federated foundation model training through shared low-rank projections and dynamic subspace updates. The method is practically motivated. The paper also analyzes the aggregation bias arising from projection disagreement and data heterogeneity, providing theoretical support through bias bounds and conditional convergence guarantees for projected SGD.
- FedLore was evaluated on a range of vision and language tasks, including LLaMA pre-training at different model scales. The broad experimental evaluation demonstrates competitive performance of the proposed method.
Critical weaknesses
- FedSub [1] and FedSLoP [2] have already explored shared and dynamically updated low-dimensional subspaces in federated learning. Although FedLore differs in subspace selection and communication mechanisms, the lack of discussion and direct comparisons makes its additional contributions unclear.
- The nonconvex convergence guarantee relies on the Aggregate global-gradient coverage assumption (Assumption 4), which is not guaranteed by the implemented subspace selection strategy. Moreover, the analysis focuses on projected SGD, whereas the experiments use AdamW, limiting the theoretical support for the actual algorithm.
References
[1] Zhang, J., Fan, Y., Xu, Y., & Yuan, K. (2026). An efficient subspace algorithm for federated learning on heterogeneous data. IEEE Transactions on Signal Processing.
[2] He, Y., Huang, Z., Geng, J., & Yuan, K. (2026). FedSLoP: Memory-Efficient Federated Learning with Low-Rank Gradient Projection. arXiv preprint arXiv:2604.24012.
Questions for authors
- What are FedLore's key contributions relative to FedSub and FedSLoP, particularly in subspace construction and model synchronization?
- Can the nonconvex convergence guarantee in Theorem 2 (established for projected SGD) be extended to AdamW, which is used in the actual implementation?
- The experiments only consider full client participation, whereas partial participation is more common in practical federated learning. How does partial participation affect FedLore's model performance, synchronization, and communication efficiency?