acceptodds
Under review as a conference paper at ICLR 2027

Overcoming the Communication-Performance Tradeoff in LLM Pretraining

Abstract

Communication-efficient distributed training algorithms (e.g., DiLoCo) have received considerable interest due to their benefits for training large language models (LLMs) in bandwidth-constrained settings, such as across datacenters and over the internet. While these local-update methods achieve communication reduction through reduced synchronization, they still require communicating dense model-sized pseudo-gradients, resulting in a communication bottleneck even for cross-datacenter links. While quantization is often applied to reduce the pseudo-gradient's size, in the context of LLM pre-training, existing approaches have not been able to leverage sparsification without incurring significant performance degradation. In this work, we introduce SparseLoCo, a communication-efficient training algorithm for LLMs that can effectively leverage Top-k sparsification and 2-bit quantization to reach extreme sparsity in the communicated pseudo-gradient, as high as 97-99%, while achieving lower final loss than dense DiLoCo. In our empirical study of language model pre-training, we demonstrate that SparseLoCo's improvements over DiLoCo in performance and pseudo-gradient compression are maintained across dense model scales (178M-2B), an MoE transformer (645M-A273M), increasing the number of workers, and increasing communication intervals.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM, LLM pre-training, distributed training, communication efficiency, gradient compression, Top-k sparsification, error feedback, LocalSGD, DiLoCo

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_mQp2OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposed a communication-efficient LLM pertaining method (SparseLoCo) with top-k sparsification and 2-bit quantization. The experiments showed that SparseLoCo training maintain competitive loss with far more less communication.

Critical strengths
  1. The paper addresses an important problem in the scenario of distributed LLM training with communication limitation.
  2. The experiments cover different aspects such as model size, communication interval, worker count, sparsity, quantization, and MoE training, which provide a solid evidence of the proposed method.
Critical weaknesses
  1. The author needs to be clear about the concrete contribution as I think Local updates, sparsification, quantization, and error feedback have already been well studied.
  2. The ablations support individual design choices, but the paper provides neither a clear account of how the combined components produce the gains nor a convergence guarantee for SparseLoCo.
  3. The fairness of the communication comparison is unclear as the proposed method use 2-bit and other remains 8-bit.
Questions for authors
  1. Could you please explain the key difference between proposed method and QSparse-local-SGD beyond using AdamW for local updates?
  2. Could you please provide more evidence about that error feedback plays the role of outer momentum?
  3. Are there any conditions under which the proposed method is provide to be converge?
  4. How does the baseline methods perform with 2-bit quantization?
  5. Could you please run the experiment with more seeds and report the std?
3 comments

Dear Reviewer mQp2,

Thank you for your detailed review and for recognizing the importance of the problem and that our experiments provide solid evidence for the proposed method. We respond to each point below. All corresponding changes are marked in blue in the revision.


W3/Q4: Baselines at 2 bits

We used 8-bit for the baselines because this is the precision commonly used in practice [3.6] and at which performance degradation is known to be avoided. Nevertheless, even if the baselines used the same 2-bit quantization, SparseLoCo would still communicate much less than both.

To demonstrate the performance degradation of the baselines at 2-bit quantization, we evaluated DiLoCo and DeMo with 2-bit quantization, following the main experimental setup of Table 1. For DiLoCo, even with error feedback, we observe a loss degradation from 2.76 to 2.79. DeMo with 2-bit quantization also degrades, from 2.83 to 2.85. In contrast, SparseLoCo shows no degradation at 2 bits compared to full precision (Table 5). We have updated Table 1 accordingly.


Q2: Error feedback plays the role of outer momentum

To make the connection between aggressive sparsity + EF and momentum more visible, consider Top-kk EF with β≤1\beta \leq 1: ut=βet−1+Δtu_t = \beta e_{t-1} + \Delta_t, Δ^t=TopK(ut)\hat{\Delta}_t = \mathrm{TopK}(u_t), et=ut−Δ^te_t = u_t - \hat{\Delta}_t. Fix a coordinate ii. If it is selected at outer step tt, then Δ^t,i=ut,i\hat{\Delta}_{t,i} = u_{t,i} and et,i=0e_{t,i} = 0. If it is not selected, then Δ^t,i=0\hat{\Delta}_{t,i} = 0 and et,i=ut,i=βet−1,i+Δt,ie_{t,i} = u_{t,i} = \beta e_{t-1,i} + \Delta_{t,i}. Thus, unselected coordinates accumulate past pseudo-gradients, while selected ones are transmitted and reset. If ri(t)r_i(t) is the most recent round at or before tt when coordinate ii was reset, then ut,i=∑s=ri(t)+1tβ,t−sΔs,iu_{t,i} = \sum_{s=r_i(t)+1}^{t} \beta^{\\,t-s} \Delta_{s,i}. Therefore, viewed coordinate-wise, there is a weighted sum of past pseudo-gradients since the last reset of that coordinate. Furthermore, in most experiments we use β<1\beta < 1, so this is an exponentially weighted sum. Therefore, viewed coordinate-wise, Top-kk + EF can behave like a heavy-ball momentum with stochastic resets, which become infrequent with high sparsity. We emphasize that this view does not apply to other compressors (such as quantization), where each transmitted coordinate is typically non-zero. Practically, for the most effective cases (e.g., 3% density), we observed resets on average every 35 outer steps for a given coordinate, a significant number of outer steps. We acknowledge that this is not classical momentum because of the resets.

Empirically, in Section 4.2 we study DiLoCo-LOM-Sub-kk, where the largest-magnitude entries of the local momentum are removed after each synchronization, mirroring the error feedback in Top-kk compression (L290–296). These local accumulators remain closely aligned with a reference global momentum (L306–312), with no loss in performance (Table 2). During the rebuttal, we extended this to an actual SparseLoCo run: using the 178M setting (R=8R=8, H=50H=50, 3.12% density), we compare the aggregate of the replicas' outer EF buffers to a reference outer momentum over 50 outer steps and observe a cosine similarity of 0.788 on average, which confirms that the error-feedback buffers of SparseLoCo itself are strongly aligned with an outer momentum. We have added this experiment to Section 4.2, and the discussion above to Appendix W.

Q5: Multiple-seed runs and standard deviations

We ran SparseLoCo with three different seeds (42, 43, 44), each determining both the model initialization and the data order, following Table 1's setting at 3.12% density:

Seed 42Seed 43Seed 44Mean ± std
Final loss2.69902.69952.69552.6980 ± 0.0022

The standard deviation across seeds is 0.0022, with a 95% confidence interval of [2.6926, 2.7034] for the mean. The variation across seeds is very small, and far smaller than the 0.06 gap to DiLoCo in Table 1 (2.70 vs. 2.76). We have added these results to the revision (Appendix T) and updated the Limitations section (Appendix S) accordingly. We note that, due to the cost of LLM pre-training and the typical low variance at large scale (often below 0.01 validation), single-run comparisons are the standard protocol in this literature [3.2, 3.3, 3.4, 3.5, 3.7, 3.8, 3.9, 3.10].


Q3: Convergence guarantee

As requested we have added a convergence guarantee for a simplified version of SparseLoCo to the revision (Appendix X).


W1/Q1: Concrete contributions and differences from QSparse-local-SGD

We agree that local updates, sparsification, quantization, and error feedback have each been studied, largely from a theoretical perspective. However, practical versions of these algorithms and of their combination remain an open question. Our Related Work (L130–137) cites QSparse-local-SGD [3.1] as one of the few works that combine multi-iteration methods such as LocalSGD with error feedback, but it is worth emphasizing that there are significant differences to our work, which we highlight below. It focuses on a generic treatment of local SGD and broad set of compressors, showing theoretically and empirically that convergence rates and final performance can be maintained or only slightly degraded for light CNN tasks. We present a practical algorithm (aiming to be competitive with state-of-the-art Data Parallel training) tailored for LLM pre-training, that incorporates adaptive inner optimizers, modern learning rate schedules, and the specific combination of a particular aggressive Top-k compression operator (applied chunk-wise) and a decaying error-feedback term. Most importantly, we show, empirically, that error feedback, when combined with aggressive sparsity in the SparseLoCo setting, can speed up convergence and improve final performance compared to the same local optimizer without error feedback (Figure 3). To the best of our knowledge, this novel observation is neither reported in the literature nor supported by the theory or experiments of Basu et al. (which suggest that it can at best preserve performance compared to a LocalSGD optimizer without outer momentum). Finally, we note that the DiLoCo line of work [3.2, 3.5] focuses on obtaining practical results in the LLM setting and often builds on the large body of literature on local optimization. This literature treats the Nesterov outer momentum as crucial for DiLoCo [3.2–3.5]. Naively adding Top-k and error feedback to DiLoCo degrades its performance (Figure 1, Table 18), and one would expect a degradation from removing the outer momentum (Table 2 and [3.2]). While SparseLoCo outperforms DiLoCo, the analysis of QSparse-local-SGD would suggest that SparseLoCo at best matches DiLoCo without outer momentum. Following your comment, we have updated the discussion of QSparse-local-SGD in the Related Work in the revision to state this difference explicitly.

W2: Why SparseLoCo improves performance and convergence guarantee

First, we connect error feedback to the outer momentum in Section 4.2 and in our response to Q2, and provide an intuition for why SparseLoCo, which has no outer momentum, is able to match DiLoCo (in Section 4.4, L389-403). As for the improvements over DiLoCo, we provide a plausible explanation connecting sparse aggregation in SparseLoCo at a well-chosen k with model-merging methods (L405-408). Table 5 (left) also shows the importance of Top-k, supporting our hypothesis that the gains come from selecting high-saliency coordinates. Overall, we agree this is a heuristic suggestion (and indicate it as a hypothesis) and does not provide a full, theoretically satisfying explanation. That said, we are confident our experiments demonstrate the improvement and include code and hyperparameter search grids. We note that the performance of DiLoCo variants such as using Nesterov outer momentum over Polyak or outer momentum itself is not fully understood in the literature; thus, the tools for analyzing these methods are limited, and a fair standard of proof should be applied. On the other hand, this is an algorithm that has been studied and extended in top-tier venues (e.g., NeurIPS spotlight [3.4], COLM [3.3], ICML [3.5], ICLR [3.8]). Ultimately, we believe the importance of the Communication Efficient LLM pretraining problem to be high and thus, in turn, our contribution of proposing an algorithm that is at the Pareto frontier of existing methods on this problem is significant. We have added a convergence guarantee for a simplified version of SparseLoCo to the revision (Appendix X).


We hope these clarifications and the new experiments directly address your concerns. If you feel these updates strengthen the paper and resolve your concerns, we kindly ask you to consider raising your score.

References.

[3.1] Basu et al., Qsparse-local-SGD: Distributed SGD with Quantization, Sparsification, and Local Computations, 2019.

[3.2] Douillard et al., DiLoCo: Distributed Low-Communication Training of Language Models, 2023.

[3.3] Douillard et al., Streaming DiLoCo with Overlapping Communication: Towards a Distributed Free Lunch, 2025.

[3.4] Charles et al., Communication-Efficient Language Model Training Scales Reliably and Robustly: Scaling Laws for DiLoCo, 2025.

[3.5] Thérien et al., MuLoCo: Muon Is a Practical Inner Optimizer for DiLoCo, arXiv:2505.23725v1, 2025.

[3.6] Jaghouar et al., INTELLECT-1 Technical Report, 2024.

[3.7] Peng et al., DeMo: Decoupled Momentum Optimization, 2024.

[3.8] Cheng et al., EDiT: A Local-SGD-Based Efficient Distributed Training Method for Large Language Models, 2025.

[3.9] Jaghouar et al., OpenDiLoCo: An Open-Source Framework for Globally Distributed Low-Communication Training, 2024.

[3.10] Liu et al., Asynchronous Local-SGD Training for Language Modeling, 2024.

Reviewer_wY1NOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies compressing the pseudo-gradients exchanged in local-update language-model training. SparseLoCo combines infrequent synchronization, chunk-wise Top-k sparsification, quantization, and a local error-feedback buffer replacing the global outer momentum. The evaluation covers dense and MoE language models, communication intervals and worker counts, some component ablations, a bandwidth-limited wall-clock experiment, and a large-scale deployment.

Critical strengths

S1. The large-scale, real-world deployment is a strong, positive aspect of the paper.

S2. The observation that the preferred sparsity changes with the number of inner steps is very interesting.

Critical weaknesses

W1. I am currently unsure whether the reported differences establish the claimed performance advantage with sufficient confidence. This matters a lot for interpreting the loss and downstream-accuracy comparisons in Tables 2–4.

W2. The scope of the claims needs to be much clearer and more explicit. Is the contribution limited to the LLM training settings developed and evaluated in this work, or is it intended to apply to LLM settings generally? And what makes that the case? This distinction is critical to understanding what the paper establishes and evaluating its contributions and impact.

Questions for authors

Q1. What is the statistical confidence in the reported differences in Tables 2–4, and how many additional downstream examples are answered correctly? When the performance gaps are so limited, a deeper analysis is strongly recommended.

Q2. Can the authors state clearly which claims apply to the particular LLM settings evaluated here, and which are intended to hold more generally?

3 comments

Dear Reviewer wY1N,

Thank you for your thoughtful assessment, and for highlighting the large-scale real-world deployment and the link between the preferred sparsity and the number of inner steps. We respond to each point below. All corresponding changes are marked in blue in the revision.


W1: Consistency of gains over DiLoCo

We would first like to highlight that the advantage is consistent and not confined to a single configuration. Across model scales (178M, 512M, 2B), replica counts R∈8,16,32R \in \\{8, 16, 32\\}, communication intervals HH, and three architectures (LLaMA-style dense, GPT-2, and a Qwen2-style MoE), SparseLoCo matches or outperforms DiLoCo in every setting. We also re-emphasize that SparseLoCo is designed to reduce the communication volume. Even at matched performance, its communication efficiency is already a substantial practical improvement for LLM training at scale.


Q1: Statistical confidence and downstream details

Regarding statistical confidence, we have added a new multi-seed experiment to the revision. Using the main setting of the paper (Table 1, 3.12% density), we trained SparseLoCo with three different seeds (42, 43, 44), each determining both the model initialization and the data order:

Seed 42Seed 43Seed 44Mean ± std
Final loss2.69902.69952.69552.6980 ± 0.0022

We see very minimal variation across seeds: the standard deviation is 0.0022, and the 95% confidence interval of the mean is [2.6926, 2.7034]. For comparison, the gap to DiLoCo in Table 1 is 0.06 (2.70 vs. 2.76), far larger than this seed-to-seed variation. We have added a new section to the appendix for this experiment (Appendix T), with a mention in the main text (Section 4), and have updated the Limitations section (Appendix S) accordingly. We note that, due to the cost of LLM pre-training, single-run comparisons are the standard protocol in this literature [2.1, 2.2, 2.3, 2.4, 2.6, 2.8, 2.10, 2.16]. Moreover, at this scale of data, the variation across seeds is typically low, as our three-seed experiment above shows (standard deviation of 0.0022).

We have added the downstream details for Tables 3 and 4 to the revision (Appendix U): for each benchmark, the number of questions and the number answered correctly by each method. At 512M, SparseLoCo outperforms DiLoCo on every benchmark, answering 228 more examples correctly in total, which is a statistically significant difference (z-test, p<0.01p < 0.01). At 2B, the two methods are close, with SparseLoCo slightly ahead (77 more examples in total). We also re-emphasize that SparseLoCo is designed to reduce the communication volume. Even at matched performance, its communication efficiency is already a substantial practical improvement for LLM training at scale. Note that due to the cost of training at this scale, we did not search over the sparsity level for the 2B run and used a single density.


W2/Q2: Scope of the claims

Our work studies communication-efficient LLM pre-training in the low-bandwidth setting of DiLoCo [2.1, 2.2], which is now a well-established area of research that has received significant and growing interest in recent years [2.2–2.15]. Within this setting, SparseLoCo requires no architecture-specific modification and can be directly applied to any architecture or new pre-training dataset. Our evaluation supports this claim across three architectures and settings. We have made this scope explicit in the introduction and in the Limitations section (Appendix S) of the revision.


We hope these clarifications and the new multi-seed experiment directly address your concerns. If you feel these updates strengthen the paper and resolve your concerns, we kindly ask you to consider raising your score.

References.

[2.1] Douillard et al., DiLoCo: Distributed Low-Communication Training of Language Models, 2023.

[2.2] Charles et al., Communication-Efficient Language Model Training Scales Reliably and Robustly: Scaling Laws for DiLoCo, 2025.

[2.3] Douillard et al., Streaming DiLoCo with Overlapping Communication: Towards a Distributed Free Lunch, 2025.

[2.4] Thérien et al., MuLoCo: Muon Is a Practical Inner Optimizer for DiLoCo, 2025.

[2.5] Hewa Koneputugodage et al., Factored Gossip DiLoCo: Reducing Blocking Communication in DiLoCo, 2026.

[2.6] Cheng et al., EDiT: A Local-SGD-Based Efficient Distributed Training Method for Large Language Models, 2025.

[2.7] Iacob et al., MT-DAO: Multi-Timescale Distributed Adaptive Optimizers with Local Updates, 2026.

[2.8] Jaghouar et al., OpenDiLoCo: An Open-Source Framework for Globally Distributed Low-Communication Training, 2024.

[2.9] Jaghouar et al., INTELLECT-1 Technical Report, 2024.

[2.10] Peng et al., DeMo: Decoupled Momentum Optimization, 2024.

[2.11] Khaled et al., Understanding Outer Optimizers in Local SGD: Learning Rates, Momentum, and Acceleration, 2025.

[2.12] Jovanović et al., LoRDO: Distributed Low-Rank Optimization with Infrequent Communication, 2026.

[2.13] Kale et al., Eager Updates for Overlapped Communication and Computation in DiLoCo, 2025.

[2.14] Kolehmainen et al., NoLoCo: No-all-reduce Low Communication Training Method for Large Models, 2025.

[2.15] Qi et al., DiLoCoX: A Low-Communication Large-Scale Training Framework for Decentralized Cluster, 2025.

[2.16] Liu et al., Asynchronous Local-SGD Training for Language Modeling, 2024.

Reviewer_wY1NOpenReview ↗

Thanks for the rebuttal. I will raise my rating.

Dear Authors,

I appreciate your rebuttal, and I find it satisfactory in resolving my main concerns. I'll recommend acceptance. I acknowledge other reviewers' concerns related to the theoretical depth of the EF analysis and the ablation experiments on some components of the method (and their interpretation). These are important concerns, but I believe this work contributes sufficiently to the area to warrant acceptance.

Reviewer_ByqTOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies LLM pretraining under bandwidth constraints. SparseLoCo performs H local AdamW steps per worker, forms a pseudo-gradient from the parameter difference, adds a decayed local error-feedback buffer, and communicates a chunk-wise Top-k, 2-bit-quantized update. Compression residuals remain local, while the aggregated update directly updates the model, replacing DiLoCo's global Nesterov outer update. At suitable densities, the paper reports lower communication and final loss: in the main 512M setting, 3.12% density yields loss 2.70 versus DiLoCo's 2.76, with pseudo-gradient message size reduced from 512.40 MB to 17.21 MB. Experiments also cover model scales, MoE, communication intervals, and replica counts.

Critical strengths
  1. The main controlled results are practically meaningful, and the communication accounting is consistent. Table 1 shows that, for 512M parameters, approximately 10.26B tokens, R=8, and H=15, SparseLoCo at 3.12% density achieves loss 2.70, close to AdamW DDP's 2.69 and better than DiLoCo's 2.76; both local-update methods synchronize 163 times. The 17.21 MB message size is consistent with the index-coding costs in Appendix J plus 2-bit values. The MoE results in Table 8 provide further support: 2.741 versus DiLoCo's 2.803 and DDP's 2.739.

  2. The method offers a practically useful, incremental contribution, supported by comparisons with closely related variants. Qsparse-local-SGD already combines local computation, sparsification, quantization, and error compensation, while DeMo uses local momentum buffers for error feedback. The methodological novelty therefore lies primarily in adapting the update rule to multi-step AdamW pseudo-gradients and empirically examining the interaction between the additional Nesterov outer update and aggressive sparsification. It does not introduce a new sparsification or quantization principle, so the technical novelty is incremental. Nevertheless, adapting these mechanisms to multi-step LLM pretraining and obtaining a better loss–communication tradeoff remains a meaningful contribution. The comparisons in Figure 1 and Table 18 support this design choice: at 178M, R=8, H=15, and 3.12% density, SparseLoCo achieves 2.91 versus 3.15 for compressing local Nesterov directions and 2.97 for dense DiLoCo. Other ablations examine Random-k, quantization, and DCT/chunking.

  3. The paper is generally clear and easy to follow, progressing logically from DiLoCo to LOM and SparseLoCo, with a clear correspondence between equations and algorithms and well-organized figures, tables, and appendices.

Critical weaknesses
  • The robustness of the gains across model scales and communication intervals is not yet sufficiently established. In Table 4, the 2B loss improves only from 2.37 to 2.36, and downstream results are mixed; for example, ARC-Easy is 60.48% for DiLoCo versus 59.05% for SparseLoCo. The authors explicitly acknowledge the absence of error bars or confidence intervals, which limits interpretation of small differences. According to Appendix R, the 2B DiLoCo inner learning-rate search includes only 6e-4 and 8e-4, with the optimum at the upper boundary; SparseLoCo searches 8e-4 and 1e-3 and selects 1e-3. Experiments at other H values mainly reuse parameters tuned at H=15. These facts neither establish unfair baseline tuning nor negate the observed 512M/MoE gains, but they leave uncertainty about improvements under configuration transfer versus advantages that would persist after more extensive tuning.

  • The mechanistic explanation for actual SparseLoCo remains primarily supported by indirect evidence. The authors explicitly interpret the EF state as a local approximation of outer momentum. Proposition 1 correctly establishes equivalence between uncompressed LOM and global momentum, but the similarity diagnostic in Section 4.2 concerns LOM-Sub-k (25%) accumulators over only the first 20 outer steps. It does not directly establish whether the EF states of actual SparseLoCo retain this resemblance to outer momentum at operating densities and later in training, or how much this connection explains the performance gains. I recognize that the authors already frame parts of this explanation as hypotheses; more clearly separating the intuition about state approximation from a mechanistic explanation of the gains would strengthen the exposition. This limits the depth of the mechanistic insight without negating the reported empirical results, and does not call for equivalence of update directions or a complete convergence theorem.

Questions for authors
  1. Regarding W1, are there existing 2B DiLoCo results at inner LR=1e-3, or a reason for excluding this setting? If repeated runs or neighboring-hyperparameter results exist for representative 512M/2B settings, could they clarify the stability of the gains? Please also distinguish configuration transfer from separate tuning at other H values. This would help establish the magnitude and scope of the advantage without requiring another large-scale deployment run.
  2. Regarding W2, since the approximation concerns the EF state, do measurements for actual SparseLoCo at its operating densities support this interpretation beyond the first 20 outer steps? If such evidence is unavailable, could the authors clarify which parts of the mechanistic explanation remain hypotheses?
2 comments

Dear Reviewer ByqT,

Thank you for your careful review and for recognizing that our controlled results are practically meaningful and that the paper is clear and easy to follow. We respond to each point below. All corresponding changes are marked in blue in the revision.


W1/Q1: Repeated runs and tuning across H

During the rebuttal, we ran SparseLoCo with three different seeds (42, 43, 44), each determining both the model initialization and the data order, following Table 1's setting at 3.12% density:

Seed 42Seed 43Seed 44Mean ± std
Final loss2.69902.69952.69552.6980 ± 0.0022

The standard deviation across seeds is 0.0022, with a 95% confidence interval of [2.6926, 2.7034] for the mean. The variation across seeds is very small, and far smaller than the 0.06 gap to DiLoCo in Table 1 (2.70 vs. 2.76). We have added these results to the revision (Appendix T) and updated the Limitations section (Appendix S) accordingly. We note that, due to the cost of LLM pre-training, single-run comparisons are the standard protocol in this literature [1.1–1.8]. Moreover, at this scale of data, the variation across seeds is typically low, as our three-seed experiment above shows (standard deviation of 0.0022).

We agree that a broader search for the baselines at higher HH makes the comparison more conclusive. Following your advice, we extended the DiLoCo search at H=50H=50 (512M, R=8R=8, the setting of Table 6, first row) to outer learning rates in 0.4,0.6,0.8,1.0\\{0.4, 0.6, 0.8, 1.0\\} and inner learning rates in 6×10−4,8×10−4,10−3\\{6\times10^{-4}, 8\times10^{-4}, 10^{-3}\\}. The setting found optimal at H=15H=15 (αinner=8×10−4\alpha_{\mathrm{inner}} = 8\times10^{-4}, αouter=0.6\alpha_{\mathrm{outer}} = 0.6) remains optimal at H=50H=50, so the configuration transfer did not disadvantage DiLoCo. Due to the limited time of the rebuttal period, we did not repeat this search for SparseLoCo, whose H=50H=50 results therefore still use the transferred H=15H=15 settings, while still outperforming DiLoCo (Table 6, first row: 2.78 vs. 2.81). We have included this experiment in Appendix V in the revision.

DiLoCo, 512M, H=50H=50inner LR 6e-4inner LR 8e-4inner LR 1e-3
outer LR 0.42.902.872.86
outer LR 0.62.852.812.82
outer LR 0.82.822.812.82
outer LR 1.02.822.822.83

W1/Q1: 2B search grid and performance

At 2B, due to the computational cost, we used a more restricted hyperparameter sweep. The search space is not an independent search whose optimum happens to fall on a boundary; it transfers the settings established at smaller scale experiments. At 178M and 512M, where the sweeps are exhaustive (Table 19), the optimal inner learning rate is 8×10−48\times10^{-4} for DiLoCo (observing DiLoCo’s performance degrades at 10−310^{-3}) and 10−310^{-3} for SparseLoCo. Since optimal learning rates typically decrease with model size (Figure 1 in [1.9, 1.10]), we searched only learning rates at or below these optima at the 2B scale, as discussed in Appendix R (L1397-1399), which we have slightly revised to improve clarity.

We agree that at 2B the differences in Table 4 are smaller, but they still show significant benefit. We emphasize that even with closer validation performance, SparseLoCo provides a large communication benefit over DiLoCo. Note that due to the cost of training at this scale, we did not search over the sparsity level for the 2B run and used a single density. More broadly, the advantage is consistent and not confined to a single configuration: across model scales (178M, 512M, 2B), replica counts R∈8,16,32R \in \\{8, 16, 32\\}, communication intervals HH, and three architectures (LLaMA-style dense, GPT-2, and a Qwen2-style MoE), SparseLoCo matches or outperforms DiLoCo in every setting. We also re-emphasize that SparseLoCo is designed to reduce the communication volume. Even at matched performance, its communication efficiency is already a substantial practical improvement for LLM training at scale.


W2/Q2: EF state as a local approximation of outer momentum

Regarding the 20 outer steps, we note that the accumulator's effective horizon is 1/(1−β)=101/(1-\beta) = 10 outer steps (β=0.9\beta = 0.9 in DiLoCo), so the first 20 outer steps already span the buffer's memory and a longer window is not expected to change the result. We have now extended the measurement to 50 outer steps and observe the same behavior as reported in Section 4.2 (L306-312): the local accumulators remain a strong directional proxy for the global reference accumulator.

As you suggest, we have added an experiment that compares the EF state of actual SparseLoCo directly with a reference outer momentum at its operating density. We ran SparseLoCo at the 178M scale with R=8R=8, H=50H=50, and density 3.12%, maintaining a reference momentum over aggregated uncompressed pseudo-gradients. Over the same 50-outer-step horizon as above, we compute the cosine similarity between the aggregated error-feedback buffers and the reference outer momentum, averaging 0.788, in line with the ≥0.75\geq 0.75 reported for the DiLoCo-LOM-Sub-kk accumulators in Section 4.2 (L306–312). We have added this measurement to Section 4.2, with the details in Appendix W.

The above shows the connection between error feedback in SparseLoCo and the outer momentum in DiLoCo, and Section 4.4 (L389-403) provides an intuition for why SparseLoCo, which has no outer momentum, is able to match DiLoCo. Regarding why SparseLoCo improves DiLoCo, we provide a plausible explanation connecting sparse aggregation in SparseLoCo at a well-chosen k with model-merging methods (L405-408). Table 5 (left) also shows the importance of Top-k, supporting our hypothesis that the gains come from selecting high-saliency coordinates. Overall, we agree this is a heuristic suggestion (and indicate it as a hypothesis) and does not provide a full, theoretically satisfying explanation. That said, we are confident our experiments demonstrate the improvement and include code and hyperparameter search grids. We note that the performance of DiLoCo variants such as using Nesterov outer momentum over Polyak or outer momentum itself is not fully understood in the literature; thus, the tools for analyzing these methods are limited, and a fair standard of proof should be applied. On the other hand, this is an algorithm that has been studied and extended in top-tier venues (e.g., NeurIPS Spotlight [1.3], COLM [1.2], ICML [1.4], ICLR [1.6]). Ultimately, we believe the importance of the Communication-Efficient LLM pretraining problem to be high and thus, in turn, our contribution of proposing an algorithm that is at the Pareto frontier of existing methods on this problem is significant. We have added a convergence proof for a simplified version of SparseLoCo to the revision in Appendix X.


We hope these clarifications and the new experiments directly address your concerns. If you feel these updates strengthen the paper and resolve your concerns, we kindly ask you to consider raising your score.

References.

[1.1] Douillard et al., DiLoCo: Distributed Low-Communication Training of Language Models, 2023.

[1.2] Douillard et al., Streaming DiLoCo with Overlapping Communication: Towards a Distributed Free Lunch, 2025.

[1.3] Charles et al., Communication-Efficient Language Model Training Scales Reliably and Robustly: Scaling Laws for DiLoCo, 2025.

[1.4] Thérien et al., MuLoCo: Muon Is a Practical Inner Optimizer for DiLoCo, 2025.

[1.5] Peng et al., DeMo: Decoupled Momentum Optimization, 2024.

[1.6] Cheng et al., EDiT: A Local-SGD-Based Efficient Distributed Training Method for Large Language Models, 2025.

[1.7] Jaghouar et al., OpenDiLoCo: An Open-Source Framework for Globally Distributed Low-Communication Training, 2024.

[1.8] Liu et al., Asynchronous Local-SGD Training for Language Modeling, 2024.

[1.9] Yang et al., Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer, 2022.

[1.10] Zhou et al., How to Set the Learning Rate for Large-Scale Pre-training?, 2026.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.