- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes LACCO, a local optimization method designed to overlap communication with computation in distributed LLM training. The key idea is to alternate between two model states and two local optimization stages, allowing global communication to proceed in parallel with local computation while avoiding stale outer updates and maintaining consistency across replicas. The paper also provides convergence guarantees for smooth non-convex objectives. Experiments on LLM pre-training show that LACCO generally outperforms existing communication-overlapping variants of DiLoCo and remains close to synchronous DiLoCo across different model sizes, communication intervals, and numbers of replicas.
Critical strengths
-
Despite some limitations, I found the paper overall insightful. The proposed approach provides an interesting perspective on overlapping communication with local computation while avoiding stale outer updates.
-
The method is technically well motivated. The two-stage design maintains replica consistency while allowing communication and computation to proceed concurrently.
-
The paper provides theoretical convergence guarantees, which strengthens the technical foundation of the proposed method.
Critical weaknesses
- I think that the current overlap setting seems somewhat idealized compared with real-world distributed training. In practice, communication time can vary due to network congestion, bandwidth fluctuations, or stragglers. If communication does not finish within the local computation stage, workers still need to wait for it to complete, which reduces the benefit of communication-computation overlap.
- I understand that pretraining large language models is computationally expensive and that scaling up the experiments is not trivial. However, the largest model evaluated in this paper has only 325M parameters, which is still relatively small compared with modern LLMs. It would be helpful to include experiments on billion-scale models if time and computational resources permit, although I would not consider the lack of such experiments a factor that affects my overall rating of the paper.
- The extra memory and data-transfer overhead may become more significant when scaling LACCO to much larger models. The paper suggests offloading the inactive optimizer states to CPU memory to reduce GPU memory usage, which works reasonably well at the evaluated scale. Yet, modern LLMs can contain tens of billions of parameters or more, where both the additional optimizer states and the corresponding CPU-GPU transfers could become substantial. It would be useful to discuss how this overhead scales to such model sizes and whether the offloading cost can still be effectively hidden by local computation.
- In line 37, the discussion of communication-efficient training is somewhat incomplete. Besides quantization and sparsification, zeroth-order methods can reduce communication using scalar directional derivatives and shared random seeds, while LoRA-based methods can communicate low-rank updates. These related directions would be worth being mentioned.
Questions for authors
See weaknesses above. I will be willing to update my rating if the authors can address my concerns.
4 comments
References:
[1] S. Jaghouar, J. M. Ong, M. Basra, et al. INTELLECT-1 Technical Report. arXiv, 2024.
[2] A. Douillard, Y. Donchev, K. Rush, et al. Streaming DiLoCo with Overlapping Communication: Towards a Distributed Free Lunch. CoLM, 2025.
[3] S. Kale, A. Douillard, and Y. Donchev. Eager Updates for Overlapped Communication and Computation in DiLoCo. arXiv, 2025. [4] Meta PyTorch. TorchFT: Fault Tolerance for PyTorch. 2026.
[5] Z. Qin, D. Chen, B. Qian, B. Ding, Y. Li, and S. Deng. Federated Full-Parameter Tuning of Billion-Sized Language Models with Communication Cost under 18 Kilobytes. ICML, 2024.
[6] Z. Li, B. Ying, Z. Liu, C. Dong, and H. Yang. Achieving Dimension-Free Communication in Federated Learning via Zeroth-Order Optimization. ICLR, 2025.
[7] H. Feng, T. Pang, C. Du, W. Chen, S. Yan, and M. Lin. BAFFLE: A Baseline of Backpropagation-Free Federated Learning. ECCV, 2024.
[8] E. J. Hu, Y. Shen, P. Wallis, et al. LoRA: Low-Rank Adaptation of Large Language Models. ICLR, 2022.
[9] Z. Wang, Z. Shen, Y. He, G. Sun, H. Wang, L. Lyu, and A. Li. FLoRA: Federated Fine-Tuning Large Language Models with Heterogeneous Low-Rank Adaptations. NeurIPS, 2024.
[10] S. Babakniya, A. R. Elkordy, Y. H. Ezzeldin, Q. Liu, K.-B. Song, M. El-Khamy, and S. Avestimehr. SLoRA: Federated Parameter Efficient Fine-Tuning of Language Models. International Workshop on Federated Learning in the Age of Foundation Models, NeurIPS, 2023.
Thank you for your thoughtful and comprehensive responses to my comments. I am satisfied with the clarifications and appreciate the authors' efforts in addressing my concerns. I am happy to raise my score to 3. I wish the authors the best of luck with the final decision!
Dear Reviewer, thank you very much for your thoughtful engagement and for raising your score!
Author response to reviewer vJLX
We sincerely thank the reviewer for their positive assessment, particularly for finding the paper insightful, well motivated and interesting, highlighting the overlap of communication and computation without stale outer updates and our theoretical convergence guarantees. We welcome the opportunity to address their concerns and further clarify our contributions.
1. Idealised Overlap
2. Experimental Scale
We appreciate the suggestion and agree that larger-scale experiments are valuable. Within our academic compute budget, we conducted additional experiments with a 0.5B model, obtaining the following results:
LACCO remains competitive with other concurrent methods, often outperforming them, particularly Delayed DiLoCo, its direct concurrent. Please also see the answer 3 to reviewer 1zdU as well as the new Table 4.
3. Offloading
4. Missing citations
We thank the reviewer for pointing this out and will add the relevant references [5–10]. In the revised manuscript, we will expand this discussion in both the Introduction (Section 1) and Related Work (Section 2) as follows:
We thank the reviewer again and would be happy to provide additional insights if necessary!