acceptodds
Under review as a conference paper at ICLR 2027

LACCO: Local Accumulation with Computation–Communication Overlap

Abstract

Training language models across low-bandwidth networks requires reducing synchronization overhead. Local optimization methods such as DiLoCo communicate only after several local steps, but outer gradient averaging and the outer update remain on the critical path. Existing computation–communication overlap methods avoid this stall through delayed or replica-local updates, which can degrade convergence or cause replicas to drift. We propose LACCO (ocal ccumulation with omputation–ommunication verlap), a two-stage algorithm that overlaps global communication with local computation while applying globally aggregated updates shared by every replica, without introducing staleness in the outer update. We establish convergence guarantees for smooth non-convex objectives. Experiments on language-model pre-training with FineWeb show that LACCO consistently outperforms communication-overlapped baselines such as Eager and Delayed DiLoCo. These gains remain robust as model size scales up to 325M parameters and across a range of communication intervals and numbers of replicas up to 16 workers, while incurring only negligible—or fully overlappable—computational or memory overhead, which demonstrates the strength of LACCO.

Primary area:
optimization
Keywords:
DiLoCo, distributed pretraining, Local SGD, LLMs

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_vJLXOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes LACCO, a local optimization method designed to overlap communication with computation in distributed LLM training. The key idea is to alternate between two model states and two local optimization stages, allowing global communication to proceed in parallel with local computation while avoiding stale outer updates and maintaining consistency across replicas. The paper also provides convergence guarantees for smooth non-convex objectives. Experiments on LLM pre-training show that LACCO generally outperforms existing communication-overlapping variants of DiLoCo and remains close to synchronous DiLoCo across different model sizes, communication intervals, and numbers of replicas.

Critical strengths
  • Despite some limitations, I found the paper overall insightful. The proposed approach provides an interesting perspective on overlapping communication with local computation while avoiding stale outer updates.

  • The method is technically well motivated. The two-stage design maintains replica consistency while allowing communication and computation to proceed concurrently.

  • The paper provides theoretical convergence guarantees, which strengthens the technical foundation of the proposed method.

Critical weaknesses
  • I think that the current overlap setting seems somewhat idealized compared with real-world distributed training. In practice, communication time can vary due to network congestion, bandwidth fluctuations, or stragglers. If communication does not finish within the local computation stage, workers still need to wait for it to complete, which reduces the benefit of communication-computation overlap.
  • I understand that pretraining large language models is computationally expensive and that scaling up the experiments is not trivial. However, the largest model evaluated in this paper has only 325M parameters, which is still relatively small compared with modern LLMs. It would be helpful to include experiments on billion-scale models if time and computational resources permit, although I would not consider the lack of such experiments a factor that affects my overall rating of the paper.
  • The extra memory and data-transfer overhead may become more significant when scaling LACCO to much larger models. The paper suggests offloading the inactive optimizer states to CPU memory to reduce GPU memory usage, which works reasonably well at the evaluated scale. Yet, modern LLMs can contain tens of billions of parameters or more, where both the additional optimizer states and the corresponding CPU-GPU transfers could become substantial. It would be useful to discuss how this overhead scales to such model sizes and whether the offloading cost can still be effectively hidden by local computation.
  • In line 37, the discussion of communication-efficient training is somewhat incomplete. Besides quantization and sparsification, zeroth-order methods can reduce communication using scalar directional derivatives and shared random seeds, while LoRA-based methods can communicate low-rank updates. These related directions would be worth being mentioned.
Questions for authors

See weaknesses above. I will be willing to update my rating if the authors can address my concerns.

4 comments
AuthorsOpenReview ↗

Author response to reviewer vJLX

We sincerely thank the reviewer for their positive assessment, particularly for finding the paper insightful, well motivated and interesting, highlighting the overlap of communication and computation without stale outer updates and our theoretical convergence guarantees. We welcome the opportunity to address their concerns and further clarify our contributions.

1. Idealised Overlap

  1. Network variability. We agree that communication delays and network variations are unavoidable in distributed training. Our approach follows standard Local SGD principles, aiming to hide communication latency by tuning the number of local gradient steps H, as developed in [2]. In fact, this limitation is inherent to all Local SGD methods and is not specific to our work. We have added the clarification on line 131 of the revised manuscript in the related work section to explicitly discuss this phenomenon
  2. Asynchronous extension. Finally, merging our framework with asynchronous methods to handle network fluctuations and stragglers is entirely possible, with no fundamental restrictions in doing so, although we consider it beyond the scope of this work.

2. Experimental Scale

We appreciate the suggestion and agree that larger-scale experiments are valuable. Within our academic compute budget, we conducted additional experiments with a 0.5B model, obtaining the following results:



ScaleMethodOverlapEval Loss ↓HellaSwag ↑PIQA ↑ARC-Easy ↑
500MDDP✗2.55837.3166.1645.50
DiLoCo✗2.55738.5366.0546.34
LACCO (Ours)✓2.561 (+0.16%)38.63 (+0.26%)66.10 (+0.08%)45.58 (−1.64%)
Eager DiLoCo✓2.576 (+0.74%)37.88 (−1.69%)66.65 (+0.91%)45.62 (−1.55%)
Delayed DiLoCo✓2.675 (+4.61%)34.12 (−11.45%)64.53 (−2.30%)42.97 (−7.27%)

LACCO remains competitive with other concurrent methods, often outperforming them, particularly Delayed DiLoCo, its direct concurrent. Please also see the answer 3 to reviewer 1zdU as well as the new Table 4.

3. Offloading

  1. Offloading challenges. We agree that offloading may pose challenges, but we would first like to emphasize that these are not specific to our method and have been widely adopted across recent DiLoCo systems, such as INTELLECT-1 [1], Streaming DiLoCo [2], and Decoupled DiLoCo [3], and are natively supported in production frameworks like TorchFT [4]. Also, in the intended practical setting, communication could be often thought as between large data centers, where memory constraints are less critical.
  2. Scalability. Regarding the scalability, we’d like to note that the times TinnerT_{\text{inner}}, TcommT_{\text{comm}}, TouterT_{\text{outer}} taken for the inner loop, communications and update of the outer loop (which includes the CPU offloading), scale linearly with the parameter size of our model, by design. We’d be happy to double check this numerically if the reviewer provide a specific model, but given the timing that we obtained (see point 5 of our response to reviewer 1zdU), where the communication is significantly larger than the other part, we believe our toy example to be a good model of a real-world example, when communication and memory is a bottleneck. Obviously there are trade-offs and the goal of this work is not to claim it is systematically better than any methods but rather to show overlapping is possible with substantial potential improvement.
  3. Robustness to local steps. We also emphasize that our method maintains better accuracy when the number of local steps increase, allowing potentially a larger substantial overlap and we added the Table 3 which has additional results. Please also see the answer 2 to reviewer 1zdU.

4. Missing citations

We thank the reviewer for pointing this out and will add the relevant references [5–10]. In the revised manuscript, we will expand this discussion in both the Introduction (Section 1) and Related Work (Section 2) as follows:

  1. In Section 1 (line 39-43): “Beyond standard compression, zeroth-order optimization methods can reduce communication costs by transmitting scalar directional-derivative estimates and reconstructing perturbation directions from shared random seeds [5, 6, 7], while LoRA-based methods communicate compact low-rank factors [8, 9, 10].
  2. In Section 2 (line 95-97): “Examples include compression (Seide et al., 2014), quantization (Alistarh et al., 2017), sparsification (Aji & Heafield, 2017), as well as low-rank updates [8, 9, 10] and zeroth-order optimization using shared random seeds [5, 6, 7].” If the reviewer has any further relevant references in mind, we would be happy to include them as well!

We thank the reviewer again and would be happy to provide additional insights if necessary!

References:

[1] S. Jaghouar, J. M. Ong, M. Basra, et al. INTELLECT-1 Technical Report. arXiv, 2024.

[2] A. Douillard, Y. Donchev, K. Rush, et al. Streaming DiLoCo with Overlapping Communication: Towards a Distributed Free Lunch. CoLM, 2025.

[3] S. Kale, A. Douillard, and Y. Donchev. Eager Updates for Overlapped Communication and Computation in DiLoCo. arXiv, 2025. [4] Meta PyTorch. TorchFT: Fault Tolerance for PyTorch. 2026.

[5] Z. Qin, D. Chen, B. Qian, B. Ding, Y. Li, and S. Deng. Federated Full-Parameter Tuning of Billion-Sized Language Models with Communication Cost under 18 Kilobytes. ICML, 2024.

[6] Z. Li, B. Ying, Z. Liu, C. Dong, and H. Yang. Achieving Dimension-Free Communication in Federated Learning via Zeroth-Order Optimization. ICLR, 2025.

[7] H. Feng, T. Pang, C. Du, W. Chen, S. Yan, and M. Lin. BAFFLE: A Baseline of Backpropagation-Free Federated Learning. ECCV, 2024.

[8] E. J. Hu, Y. Shen, P. Wallis, et al. LoRA: Low-Rank Adaptation of Large Language Models. ICLR, 2022.

[9] Z. Wang, Z. Shen, Y. He, G. Sun, H. Wang, L. Lyu, and A. Li. FLoRA: Federated Fine-Tuning Large Language Models with Heterogeneous Low-Rank Adaptations. NeurIPS, 2024.

[10] S. Babakniya, A. R. Elkordy, Y. H. Ezzeldin, Q. Liu, K.-B. Song, M. El-Khamy, and S. Avestimehr. SLoRA: Federated Parameter Efficient Fine-Tuning of Language Models. International Workshop on Federated Learning in the Age of Foundation Models, NeurIPS, 2023.

Thank you for your thoughtful and comprehensive responses to my comments. I am satisfied with the clarifications and appreciate the authors' efforts in addressing my concerns. I am happy to raise my score to 3. I wish the authors the best of luck with the final decision!

Dear Reviewer, thank you very much for your thoughtful engagement and for raising your score!

Reviewer_acEgOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

LACCO extends the two-stage ACCO mechanism to local-update training. It maintains two parameter trajectories, communicates averaged model displacements every HH inner steps, and applies shared outer updates across replicas. The paper analyzes nonconvex SGD convergence and trains 134M- and 325M-parameter models on FineWeb, with communication-interval and replica-count ablations at 134M. Its training-time estimates model overlapping computation, communication, and optimizer work.

Critical strengths
  1. The alternating trajectories create a full local phase for communication while preserving identical outer models across replicas. This avoids the replica-dependent updates used by Eager DiLoCo and offers a concrete approach to overlapping communication without introducing outer-model disagreement.
  2. With 16 replicas and H=30H=30, the 134M model reaches loss 3.154 versus Eager DiLoCo’s 3.445 under the same token budget. Together with the broader sweeps, this supports the design’s value as replica count grows, a central concern for distributed training.
Critical weaknesses

The claimed low overhead remains conditional on the timing model. An asynchronous NCCL implementation is described, but speedups are simulated. The extra optimizer-state transfers must fit within local computation; actual overlap, throughput, and peak memory are not measured.

Questions for authors

How much of LACCO’s gain over Eager DiLoCo comes from momentum averaging? On 134M at K=4K=4, H=30H=30, compare LACCO with and without averaging under a short, fixed-token budget, holding all other settings fixed. Include Eager with its reported hyperparameters and match initialization and minibatch order across runs. Report validation losses and the gaps to Eager.

1 comment
AuthorsOpenReview ↗

Author response to reviewer acEg

We sincerely thank the reviewer for their positive assessment of our work and for highlighting the key strengths of the proposed approach, particularly its ability to overlap communication with computation while maintaining consistent outer models across replicas, as well as its improved scalability and superior performance compared to Eager DiLoCo, especially as the number of replicas increases.

1. Real vs. Simulated Timings, Throughput and Memory

Thank you for pointing out this important aspect.

  1. Real Timings. As discussed in our response to reviewer 1rzdU, we conducted actual wall-clock measurements on a physical cluster with restricted inter-node bandwidth. Specifically, we ran a speed study with K = 4 workers training a 0.5B model with inter-worker bandwidths of 0.5, 1 and 5 Gbit/s and H = 30, comparing DiLoCo with two LACCO implementations: with and without CPU offload. In every configuration, LACCO fully hides the communication and the outer update, achieving speedups over DiLoCo of 1.87-1.89x at 0.5 Gbit/s, 1.44 x at 1 Gbit/s and 1.07 x at 5 Gbit/s, which is logical. For more details, please refer to point 5 of our response to reviewer 1zdU.

  2. Memory and Throughput. In the following table, we report the measured peak memory and throughput obtained under the above described scenario:

MethodPeak GPU memory0.5 Gbit/s1 Gbit/s5 Gbit/s
DiLoCo37.9 GiB (48%)77.7k tok/s103.0k tok/s138.8k tok/s
LACCO, CPU offload40.3 GiB (51%)145.5k tok/s147.8k tok/s148.4k tok/s
LACCO, fully GPU56.1 GiB (71%)146.7k tok/s148.2k tok/s148.5k tok/s

We added those results in the appendix E3 of the paper. We’d be happy to provide the timings of a specific configuration if requested by the reviewer!

2. Momentum Averaging

Thank you for raising this point!

We highlight we ran the proposed ablation on the 134M model with K = 4 and H = 30 across the full Chinchilla budget, 20,400 inner steps with a global batch size of 64 sequences. All other settings are identical to the paper, i.e. inner LR 2×10−32 \times 10^{-3}, outer LR 0.7, outer momentum 0.9 for LACCO. Moreover, all runs were executed on NVIDIA H100 GPUs strictly matching initializations, random seeds, and deterministic data ordering. Eager DiLoCo was evaluated using its reported tuned hyperparameters (inner LR 2×10−32 \times 10^{-3}, outer LR 0.4, outer momentum 0.9).

Let S0S_0 denote the inner AdamW state at the beginning of an outer iteration. We compare three inner state handling strategies, as proposed by the reviewer: (1) LACCO, no averaging (one shared state): A single AdamW state evolves sequentially through both stages (S0→SM→SLS_0 \to S_M \to S_L). The lookahead stage starts from the state left by the main stage, and its accumulated moments are carried over into the next cycle. (2) LACCO, no averaging (rewind): The lookahead stage initializes from SMS_M, but at the end of the round its state updates are discarded, and the next cycle restarts strictly from SMS_M. Thus, the lookahead trajectory never updates the optimizer state. (3) LACCO (momentum averaging): Both stages branch from S0S_0, reaching SMS_M and SLS_L, respectively. The next cycle initializes from their pairwise average, 12(SM+SL)\frac{1}{2}(S_M + S_L), applied to both first and second Adam moments.

We obtain the following results (lower is better):

Inner State HandlingValidation LossGap to Eager DiLoCoRelative Gap to Eager DiLoCo
Eager DiLoCo3.020——
LACCO, no averaging (rewind)3.029+0.009+0.30%
LACCO, no averaging (one shared state)3.008−0.012−0.40%
LACCO (momentum averaging)2.988−0.032−1.06%

As we see, even without momentum averaging, sharing the inner optimizer state across stages already outperforms Eager DiLoCo by 0.40% which is significant. Nonetheless, momentum averaging provides a substantial boost and proves to be the most effective strategy in practice, achieving the best overall loss of 2.988, which explains why we decided to use it! We would be happy to provide additional results if needed. We added those results in Appendix E2, thanks again for the suggestion.

We thank the reviewer again and would be happy to provide additional insights if necessary!

Reviewer_1zdUOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes LACCO, a two-stage local optimization method (building conceptually on things like DiLoCo and ACCO) aimed at hiding communication latency and outer-update costs in low-bandwidth, distributed LLM training. Instead of stalling while waiting for an outer gradient all-reduce, LACCO alternates between a lookahead stage and a main stage using a dual-parameter setup (θ,θ‾)(\theta, \overline{\theta}). The main pitch is that it achieves full outer-step overlap without introducing staleness or requiring replica-local correction hacks.

Critical strengths

The math and convergence proofs are solid, and the paper does a nice job showing that LACCO avoids the catastrophic drift that plagues methods like Eager DiLoCo when scaling up the number of workers (K=16K=16).

Critical weaknesses

First, Persistent Gap with Synchronous DiLoCo: The whole point of overlapping communication in Local SGD is to get a "free lunch"—hiding the network delay without hurting statistical efficiency. But across almost every experiment (Tables 2–5), LACCO consistently underperforms synchronous DiLoCo in validation loss. Because LACCO splits training into a double-stage loop, it effectively stretches out the optimization cadence or introduces drift between the two models. If I'm using DiLoCo, I'd rather just take a small synchronization stall than permanently bleed validation loss.

Second, the CPU-GPU Transfer / Memory Overhead is Brutal: To pull this off, you have to maintain 5 FP32 parameter-sized buffers, meaning a lot of heavy state shuffling between GPU and CPU memory (cLACCO=5c_{\text{LACCO}}=5 versus 2 for standard DiLoCo). According to the authors' own timing model, this pushes the outer update time up significantly, requiring H≥19H \ge 19 just to break even on overlap. In practice, unless your inner steps are quite long or your PCIe bandwidth is pristine, this memory/transfer overhead is going to eat into your training speedups.

Third, the scale is Too Small: The experiments are capped at 134M and 325M parameter models. Given that low-bandwidth distributed methods (like DiLoCo and its variants) are specifically pitched for massive multi-billion parameter pre-training regimes (1B to 7B+), stopping at 325M leaves a big question mark. We really need to see if this dual-buffer approach and the CPU offloading bottleneck hold up at scales where communication actually dominates computation.

Questions for authors

Validation Choice: You mentioned validating on θt\theta_t rather than averaging or using θ‾t\overline{\theta}_t. Did you look at how θ‾t\overline{\theta}_t performs on validation, or if the average of the two states yields a better checkpoint?

Real vs. Simulated Timings: The speedup claims lean heavily on simulated wall-clock models (Appendix D). Do you have actual wall-clock measurements on a physical cluster with restricted inter-node bandwidth (e.g., 10 Gbps)?

4 comments
AuthorsOpenReview ↗

Author response to reviewer 1zdU (1/2)

We sincerely thank the reviewer for their thoughtful and constructive feedback, and for recognizing the rigor of our theoretical analysis and proofs, as well as LACCO’s scalability as the number of workers increases. Their comments have helped us strengthen our experimental evaluation. We address the reviewer's concerns and questions below.

1. Persistent Gap with Synchronous DiLoCo

  1. Overlapping. While we agree that an unmodified synchronous method may outperform an overlapped or asynchronous method for the same number of steps, the latter offer a key system advantage: reduced wall-clock time enables more training steps and thus better final accuracy. For example, Covenant-72B [1] reports that INTELLECT-1 (10B) [2], trained with DiLoCo, incurs a synchronization overhead of 8.3 minutes on top of 38 minutes of computation, suggesting substantial potential speedups from overlapping computation and communication. In practice (see below), we observe in our experiments a speedup of about 1.9x in a low-bandwidth environment (0.5Gbit/s).
  2. Accuracy and efficiency. We emphasize that this trade-off remains an active research topic, as reflected in recent work [3-6], all from 2025-2026. Our work addresses this challenge by being the first to demonstrate effective overlapping that outperforms concurrent approaches in both final accuracy and system-level efficiency, for model sizes ranging from 0.1B to our new 0.5B trained model (see below), which degrades final accuracy for DiLoCo by at most 0.16% and improves accuracy over Eager and Delayed DiLoCo by 0.5% and 4.5% respectively for H=30, which is significant. Please also look at our nice visualization, which is helpful to plan different scenarios.

2. CPU-GPU Transfer

  1. Offloading is standard. First of all, we would like to emphasize that CPU-GPU offloading in the DiLoCo framework is not specific to our work. Indeed, it has already been successfully employed in practice by INTELLECT-1 [2], Streaming DiLoCo [3], and Decoupled DiLoCo [7]. These works build upon offloading strategies implemented for instance in TorchFT [8], which have a negligible effect on training speed for large models. We note that H=30≥19H=30 \geq 19 is a standard number of local steps, which should allow full overlap. We conducted additional experiments on offloading a 0.5B-parameter model with a bandwidth of 0.5 Gbit/s. In this setting, while the total inner computation takes approx. 51 s (H = 30 inner steps x 1.7 s each), and communication takes 50 s, fully offloading the model without any specific optimization requires only about 0.6 s.

  2. Hardware and target setting. Second, modern A100/H100 nodes (to which we have access) with PCIe Gen4 interconnects sustain CPU-GPU bandwidths of approx. 25 GB/s, while newer hardware, such as the B300, offers up to approx. 900 GB/s of bidirectional bandwidth, which is significantly better. More importantly, a usually targeted system (see https://www.coreauto.com/hitchhikers-guide-to-training-on-compute-islands) is an aggregated data center with many GPUs (e.g., 160 B300 per worker as in [9]) connected through high-bandwidth interconnects, effectively behaving as a single, tightly coupled computing system which in fact allows FSDP [10,11], so that memory should not be an issue. We note that, in this regime, the constant cLACCO=5c_{\mathrm{LACCO}} = 5 has a negligible impact given the order of magnitude of the bound. Moreover, its effect can be mitigated by increasing the number of local steps H, a setting in which LACCO remains robust compared to concurrent works.

  3. Clarification. We propose adding the following clarification at line 380: “While CPU-GPU transfers can be beneficial, our primary target setting is a tightly interconnected GPU cluster, where high-bandwidth communication enables the system to operate effectively as a single, unified computing platform. We however discuss timing in a setting where this wouldn’t be the case” and we also added the full timing in the new Table 3 of the paper. Please also check point 3 of our response to reviewer vJLX.

AuthorsOpenReview ↗

Author response to reviewer 1zdU (2/2)

3. Model Scaling

Our experimental scale is limited by our academic computing resources. Nevertheless, following the reviewer's suggestion and to the extent our compute resources permitted, we added experiments with a 0.5B model. Notably, as shown in Table 4, LACCO degrades the validation loss of DiLoCo by only 0.16%, while Eager DiLoCo and Delayed DiLoCo are respectively 0.5% and 4.5% worse than LACCO. This further validates our claims that LACCO (i) performs well in our regime and (ii) suggests that it will keep performing well when scaling the model size.

We give below the full evaluation:

ScaleMethodOverlapEval Loss ↓HellaSwag ↑PIQA ↑ARC-Easy ↑
500MDDP✗2.55837.3166.1645.50
DiLoCo✗2.55738.5366.0546.34
LACCO (Ours)✓2.561 (+0.16%)38.63 (+0.26%)66.10 (+0.08%)45.58 (−1.64%)
Eager DiLoCo✓2.576 (+0.74%)37.88 (−1.69%)66.65 (+0.91%)45.62 (−1.55%)
Delayed DiLoCo✓2.675 (+4.61%)34.12 (−11.45%)64.53 (−2.30%)42.97 (−7.27%)

We added this result in Table 4 of the paper, and please see also point 2 of our answer given to reviewer vJLX.

4. Validation choices

We thank the reviewer for this question! We compared checkpoints evaluated on θt\theta_t, θ~t\tilde{\theta}_t, and their pairwise average, observing that the three choices yield identical final validation loss. To motivate this, we tracked the L2 distance ∥θt−θ~t∥2\lVert \theta_t - \tilde{\theta}_t \rVert_2 across the entire training run, as illustrated in Figure 4 of Appendix E1. The two models start identical, drift apart during training, and eventually converge back together at the end of training. Therefore, since the two models almost coincide at the end of training, all three validation strategies yield equivalent results. We decided to include the appendix E1 that studies this distance and shows that we can freely use θt\theta_t, thank you for the suggestion!

5. Real vs. Simulated Timings

Thanks for the suggestion! We ran an experiment with K=4 workers on a 0.5B model linked to each other by low-bandwidth connections of 0.5, 1, and 5 Gbit/s. Each worker has one 80 GB A100 GPU, connected to the CPU through PCIe Gen4 x16 with bCPU/GPU≈20GB/sb_{CPU/GPU}\approx20GB/s (a standard for an academic cluster). The following table reports our finding, with TstepT_{\text{step}}, TinnerT_{\text{inner}}, TcommT_{\text{comm}}, and TouterT_{\text{outer}} denoting the time per inner step, total local-computation time, communication time, and outer-update time (including CPU offloading when applicable), respectively. TroundT_{\text{round}} denotes the measured time per communication round.

BandwidthMethodTstepT_{\text{step}} (s)TinnerT_{\text{inner}} (s)TcommT_{\text{comm}} (s)TouterT_{\text{outer}} (s)TroundT_{\text{round}} (s)Speedup
0.5 Gbit/sDiLoCo1.7251.649.40.3101.171.00×
LACCO, CPU offload1.7251.649.40.654.051.87×
LACCO, fully GPU1.7251.649.40.1553.621.89×
1 Gbit/sDiLoCo1.7251.624.80.376.381.00×
LACCO, CPU offload1.7251.624.80.653.211.44×
LACCO, fully GPU1.7251.624.80.1553.081.44×
5 Gbit/sDiLoCo1.7251.65.30.356.661.00×
LACCO, CPU offload1.7251.65.30.653.001.07×
LACCO, fully GPU1.7251.65.30.1552.951.07×

As expected, based on our timing model and the measured timings, full overlap requires at most 30 local steps at 0.5 Gbit/s, 15 at 1 Gbit/s, and 4 at 5 Gbit/s. Thus, our choice of H = 30 is sufficient to hide both communication and the outer update in all tested configurations.. Note that those results will scale with model size linearly, but we would be happy to provide additional timings if requested! We’ve added those results in Table 3.

References:

[1] J. Lidin, A. Sarfi, E. Miahi, et al. Covenant-72B: Pre-Training a 72B LLM with Trustless Peers Over-the-Internet. arXiv, 2026.

[2] S. Jaghouar, J. M. Ong, M. Basra, et al. INTELLECT-1 Technical Report. arXiv, 2024.

[3] A. Douillard, Y. Donchev, K. Rush, et al. Streaming DiLoCo with Overlapping Communication: Towards a Distributed Free Lunch. CoLM, 2025.

[4] S. Kale, A. Douillard, and Y. Donchev. Eager Updates for Overlapped Communication and Computation in DiLoCo. arXiv, 2025.

[5] C. Hewa Koneputugodage et al. Factored Gossip DiLoCo: Reducing Blocking Communication in DiLoCo. arXiv, 2026.

[6] T. Ajanthan, S. Ramasinghe, G. Avraham, et al. Asyncmesh: Fully Asynchronous Optimization for Data and Pipeline Parallelism. arXiv, 2026.

[7] Decoupled DiLoCo Team. Decoupled DiLoCo for Resilient Distributed Pre-training. arXiv, 2026.

[8] Meta PyTorch. TorchFT: Fault Tolerance for PyTorch. 2026.

[9] Core Automation. The Hitchhiker's Guide to Training on Compute Islands. 2026.

[10] Y. Zhao, A. Gu, R. Varma, et al. PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel. VLDB, 2023.

[11] S. Rajbhandari, J. Rasley, O. Ruwase, and Y. He. ZeRO: Memory Optimizations Toward Training Trillion Parameter Models. SC, 2020.

AuthorsOpenReview ↗

Additional Comment by Authors

Dear reviewer, to further address your concerns, we conducted an additional experiment at the 1B scale. A full hyperparameter sweep at this scale is beyond our computational budget, so we did not perform any additional tuning. Instead, we extrapolated the inner LRs from the 0.1B, 0.3B, and 0.5B runs via a linear fit, obtaining an inner LR of 4.5e-4 for DiLoCo and 1.5e-3 for LACCO, and trained both methods with a Chinchilla-optimal budget. Both methods reach a final validation loss of 2.400, consistent with the smaller-scale runs, where the gap between the two was already small (0.16% at 0.5B). This suggests that LACCO scales with model size. We will include these results in the revised manuscript and, resources permitting, we will run a full hyperparameter sweep at the 1B scale after the discussion period, although we expect LACCO and DiLoCo to continue performing on par! We hope this addresses your concerns, and we remain available for any further questions.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General comment for the reviewers

We would like to sincerely thank the reviewers for their insightful and constructive comments, which have helped us improve the overall quality of our work. Note that all our modifications can be visible in blue, in the main PDF. In this revised version, we have incorporated a 0.5B model and made several additional improvements, including:

  1. We run a 0.5B baseline
  2. Simplified the proofs in the main theorem, see Theorems 1 and corresponding proofs in the Appendix.
  3. Fix a minor change in Assumption 2.
  4. We also took advantage of the additional time since submission to further refine the baseline tuning of the 334M model. This led to qualitatively similar results, with slightly improved overall performance on the validation loss. The tuning methodology, consistently applied across all models, is described in Appendix B2.

Finally, we provide the following animation to let reviewers visualize the benefits of LACCO: https://anonymous.4open.science/w/visual-77EF/

Thank you again for your insightful comments and we’d be happy to answer additional comments!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.