- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposed a communication-efficient LLM pertaining method (SparseLoCo) with top-k sparsification and 2-bit quantization. The experiments showed that SparseLoCo training maintain competitive loss with far more less communication.
Critical strengths
- The paper addresses an important problem in the scenario of distributed LLM training with communication limitation.
- The experiments cover different aspects such as model size, communication interval, worker count, sparsity, quantization, and MoE training, which provide a solid evidence of the proposed method.
Critical weaknesses
- The author needs to be clear about the concrete contribution as I think Local updates, sparsification, quantization, and error feedback have already been well studied.
- The ablations support individual design choices, but the paper provides neither a clear account of how the combined components produce the gains nor a convergence guarantee for SparseLoCo.
- The fairness of the communication comparison is unclear as the proposed method use 2-bit and other remains 8-bit.
Questions for authors
- Could you please explain the key difference between proposed method and QSparse-local-SGD beyond using AdamW for local updates?
- Could you please provide more evidence about that error feedback plays the role of outer momentum?
- Are there any conditions under which the proposed method is provide to be converge?
- How does the baseline methods perform with 2-bit quantization?
- Could you please run the experiment with more seeds and report the std?
3 comments
Q5: Multiple-seed runs and standard deviations
We ran SparseLoCo with three different seeds (42, 43, 44), each determining both the model initialization and the data order, following Table 1's setting at 3.12% density:
| Seed 42 | Seed 43 | Seed 44 | Mean ± std | |
|---|---|---|---|---|
| Final loss | 2.6990 | 2.6995 | 2.6955 | 2.6980 ± 0.0022 |
The standard deviation across seeds is 0.0022, with a 95% confidence interval of [2.6926, 2.7034] for the mean. The variation across seeds is very small, and far smaller than the 0.06 gap to DiLoCo in Table 1 (2.70 vs. 2.76). We have added these results to the revision (Appendix T) and updated the Limitations section (Appendix S) accordingly. We note that, due to the cost of LLM pre-training and the typical low variance at large scale (often below 0.01 validation), single-run comparisons are the standard protocol in this literature [3.2, 3.3, 3.4, 3.5, 3.7, 3.8, 3.9, 3.10].
Q3: Convergence guarantee
As requested we have added a convergence guarantee for a simplified version of SparseLoCo to the revision (Appendix X).
W1/Q1: Concrete contributions and differences from QSparse-local-SGD
We agree that local updates, sparsification, quantization, and error feedback have each been studied, largely from a theoretical perspective. However, practical versions of these algorithms and of their combination remain an open question. Our Related Work (L130–137) cites QSparse-local-SGD [3.1] as one of the few works that combine multi-iteration methods such as LocalSGD with error feedback, but it is worth emphasizing that there are significant differences to our work, which we highlight below. It focuses on a generic treatment of local SGD and broad set of compressors, showing theoretically and empirically that convergence rates and final performance can be maintained or only slightly degraded for light CNN tasks. We present a practical algorithm (aiming to be competitive with state-of-the-art Data Parallel training) tailored for LLM pre-training, that incorporates adaptive inner optimizers, modern learning rate schedules, and the specific combination of a particular aggressive Top-k compression operator (applied chunk-wise) and a decaying error-feedback term. Most importantly, we show, empirically, that error feedback, when combined with aggressive sparsity in the SparseLoCo setting, can speed up convergence and improve final performance compared to the same local optimizer without error feedback (Figure 3). To the best of our knowledge, this novel observation is neither reported in the literature nor supported by the theory or experiments of Basu et al. (which suggest that it can at best preserve performance compared to a LocalSGD optimizer without outer momentum). Finally, we note that the DiLoCo line of work [3.2, 3.5] focuses on obtaining practical results in the LLM setting and often builds on the large body of literature on local optimization. This literature treats the Nesterov outer momentum as crucial for DiLoCo [3.2–3.5]. Naively adding Top-k and error feedback to DiLoCo degrades its performance (Figure 1, Table 18), and one would expect a degradation from removing the outer momentum (Table 2 and [3.2]). While SparseLoCo outperforms DiLoCo, the analysis of QSparse-local-SGD would suggest that SparseLoCo at best matches DiLoCo without outer momentum. Following your comment, we have updated the discussion of QSparse-local-SGD in the Related Work in the revision to state this difference explicitly.
W2: Why SparseLoCo improves performance and convergence guarantee
First, we connect error feedback to the outer momentum in Section 4.2 and in our response to Q2, and provide an intuition for why SparseLoCo, which has no outer momentum, is able to match DiLoCo (in Section 4.4, L389-403). As for the improvements over DiLoCo, we provide a plausible explanation connecting sparse aggregation in SparseLoCo at a well-chosen k with model-merging methods (L405-408). Table 5 (left) also shows the importance of Top-k, supporting our hypothesis that the gains come from selecting high-saliency coordinates. Overall, we agree this is a heuristic suggestion (and indicate it as a hypothesis) and does not provide a full, theoretically satisfying explanation. That said, we are confident our experiments demonstrate the improvement and include code and hyperparameter search grids. We note that the performance of DiLoCo variants such as using Nesterov outer momentum over Polyak or outer momentum itself is not fully understood in the literature; thus, the tools for analyzing these methods are limited, and a fair standard of proof should be applied. On the other hand, this is an algorithm that has been studied and extended in top-tier venues (e.g., NeurIPS spotlight [3.4], COLM [3.3], ICML [3.5], ICLR [3.8]). Ultimately, we believe the importance of the Communication Efficient LLM pretraining problem to be high and thus, in turn, our contribution of proposing an algorithm that is at the Pareto frontier of existing methods on this problem is significant. We have added a convergence guarantee for a simplified version of SparseLoCo to the revision (Appendix X).
We hope these clarifications and the new experiments directly address your concerns. If you feel these updates strengthen the paper and resolve your concerns, we kindly ask you to consider raising your score.
References.
[3.1] Basu et al., Qsparse-local-SGD: Distributed SGD with Quantization, Sparsification, and Local Computations, 2019.
[3.2] Douillard et al., DiLoCo: Distributed Low-Communication Training of Language Models, 2023.
[3.3] Douillard et al., Streaming DiLoCo with Overlapping Communication: Towards a Distributed Free Lunch, 2025.
[3.4] Charles et al., Communication-Efficient Language Model Training Scales Reliably and Robustly: Scaling Laws for DiLoCo, 2025.
[3.5] Thérien et al., MuLoCo: Muon Is a Practical Inner Optimizer for DiLoCo, arXiv:2505.23725v1, 2025.
[3.6] Jaghouar et al., INTELLECT-1 Technical Report, 2024.
[3.7] Peng et al., DeMo: Decoupled Momentum Optimization, 2024.
[3.8] Cheng et al., EDiT: A Local-SGD-Based Efficient Distributed Training Method for Large Language Models, 2025.
[3.9] Jaghouar et al., OpenDiLoCo: An Open-Source Framework for Globally Distributed Low-Communication Training, 2024.
[3.10] Liu et al., Asynchronous Local-SGD Training for Language Modeling, 2024.
Dear Reviewer mQp2,
Thank you for your detailed review and for recognizing the importance of the problem and that our experiments provide solid evidence for the proposed method. We respond to each point below. All corresponding changes are marked in blue in the revision.
W3/Q4: Baselines at 2 bits
We used 8-bit for the baselines because this is the precision commonly used in practice [3.6] and at which performance degradation is known to be avoided. Nevertheless, even if the baselines used the same 2-bit quantization, SparseLoCo would still communicate much less than both.
To demonstrate the performance degradation of the baselines at 2-bit quantization, we evaluated DiLoCo and DeMo with 2-bit quantization, following the main experimental setup of Table 1. For DiLoCo, even with error feedback, we observe a loss degradation from 2.76 to 2.79. DeMo with 2-bit quantization also degrades, from 2.83 to 2.85. In contrast, SparseLoCo shows no degradation at 2 bits compared to full precision (Table 5). We have updated Table 1 accordingly.
Q2: Error feedback plays the role of outer momentum
To make the connection between aggressive sparsity + EF and momentum more visible, consider Top-k EF with β≤1: ut=βet−1+Δt, Δ^t=TopK(ut), et=ut−Δ^t. Fix a coordinate i. If it is selected at outer step t, then Δ^t,i=ut,i and et,i=0. If it is not selected, then Δ^t,i=0 and et,i=ut,i=βet−1,i+Δt,i. Thus, unselected coordinates accumulate past pseudo-gradients, while selected ones are transmitted and reset. If ri(t) is the most recent round at or before t when coordinate i was reset, then ut,i=∑s=ri(t)+1tβ,t−sΔs,i. Therefore, viewed coordinate-wise, there is a weighted sum of past pseudo-gradients since the last reset of that coordinate. Furthermore, in most experiments we use β<1, so this is an exponentially weighted sum. Therefore, viewed coordinate-wise, Top-k + EF can behave like a heavy-ball momentum with stochastic resets, which become infrequent with high sparsity. We emphasize that this view does not apply to other compressors (such as quantization), where each transmitted coordinate is typically non-zero. Practically, for the most effective cases (e.g., 3% density), we observed resets on average every 35 outer steps for a given coordinate, a significant number of outer steps. We acknowledge that this is not classical momentum because of the resets.
Empirically, in Section 4.2 we study DiLoCo-LOM-Sub-k, where the largest-magnitude entries of the local momentum are removed after each synchronization, mirroring the error feedback in Top-k compression (L290–296). These local accumulators remain closely aligned with a reference global momentum (L306–312), with no loss in performance (Table 2). During the rebuttal, we extended this to an actual SparseLoCo run: using the 178M setting (R=8, H=50, 3.12% density), we compare the aggregate of the replicas' outer EF buffers to a reference outer momentum over 50 outer steps and observe a cosine similarity of 0.788 on average, which confirms that the error-feedback buffers of SparseLoCo itself are strongly aligned with an outer momentum. We have added this experiment to Section 4.2, and the discussion above to Appendix W.