- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper propose an adaptive framework AutoLoCo for distributed training. Specifically, it reduces the gradient synchronization frequency in data parallelism, with horizon controll that preserves training performance. The technical part is well written, however, the experiments and results presentation need to be improved and need clarification. Additionally, the use case of this work is not clearly defined, i.e., slow networks between GPUs where gradient communication is the bottleneck. With this assumption, "1.26 to 1.46 Gbit/s" GPU-to-GPU connection used in the experiment is not representative for modern GPU clusters (e.g., NVIDIA A100 GPUs with intra-node NVLink and inter-node infiniband/RoCEv1/RoCEv2). Furthermore, the framework claims communication efficiency for distributed LLM training. However, it is designed for data parallelism only, ignoring other communication patterns such as pipeline parallelism, fully-sharded data parallelism, expert parallelism, and tensor parallelism, which are identified as communication bottlenecks by existing work for modern LLM training. Above, I am not convinced about the generality of AutoLoCo's effectiveness, or this work has a narrow use case.
Critical strengths
- The technical part (Section 3) is well written and easy to follow.
- The implementation details are provided and contribute to the reproducibility of this work.
Critical weaknesses
-
The scope of this work is not clearly defined. AutoLoCo will be effective only if gradient communication (e.g, Allreduce) is the bottleneck in LLM training, instead of allgather/all-to-all model weights. To demonstrate its effectiveness, the experiment used a small LLM (8B) and an uncommon setting in practical system settings, i.e., 1.26 to 1.46 Gbit/s for inter-worker connections.
-
The evaluation does not validate the work's effectiveness. Only one model (LLama-3.1-8B and a 215M variant), one dataset (C5), and one baseline (DiLoCo) are used in the experiment.
Questions for authors
-
Which GPU is used for the workers, and how are GPUs connected? The hardware specifications are important for identifying the scope of this work because the use case of AutoLoCo is when GPU connections are slow. "1.26 to 1.46 Gbit/s" reported in Section 4 is not common for a practical distributed LLM training system.
-
How many more steps does AutoLoCo need to take to reach the 2.72 loss? Table 5 shows some degradation in downstream tasks with higher loss (IFEval Strict and HumanEval).
-
What is the communication pattern? Is it all-reduce by treating the workers all the same, or use a parameter server to collect the updates?
-
What is the ratio of gradient communication to the total iteration time in the experiments? Specifically, what portion of the gradient communication can be overlapped with the bucket communication in DDP, and what will the ratio be when training larger models, e.g., Llama-3.1-70B?
-
What will be the AutoLoCo performance compared to local SGD and streaming DiLoCo?
3 comments
Q2. How many extra steps did AutoLoCo need to reach 2.72, and what happened on IFEval Strict and HumanEval?
C4 training loss. We repeated the AutoLoCo training run with more optimization steps. All other training settings were the same as in the original run. The number of steps to reach 2.72 is 109,935.
Table R20. The full AutoLoCo trajectory.
| Method | 60K | 80K | 100K | 109,935 |
|---|---|---|---|---|
| AutoLoCo | 2.7716 | 2.7478 | 2.7292 | 2.7248 |
Downstream scores. IFEval Strict counts a prompt as correct only when the response satisfies every requirement, so a single missed instruction causes the whole prompt to fail. On HumanEval, a net difference of five solved problems was present on the base tests. I think that less frequent synchronization is the main reason for the performance gap. DDP synchronized gradients at each of the 6,490 optimizer steps. With AutoLoCo, each worker took consecutive steps using its local data and its own AdamW state, and the workers performed only 49 outer synchronizations over the entire run. These two update schemes lead to different parameter trajectories. Like other local update methods, AutoLoCo shows a performance gap relative to globally synchronized training, although it performs better than the other local update methods we evaluated.
Table R21. Downstream scores (%) from Tables 5 and 6.
| Metric | DDP, 8 workers | DiLoCo, 8 workers | AutoLoCo, 8 workers | DiLoCo, 16 workers | AutoLoCo, 16 workers |
|---|---|---|---|---|---|
| IFEval Strict ↑ | 37.15 | 32.72 | 32.90 | 34.01 | 34.38 |
| HumanEval+ pass@1 ↑ | 25.61 | 15.24 | 22.56 | 17.07 | 23.17 |
Table R22. Performance Gap on Downstream Tasks.
| 8-worker evaluation | DDP | AutoLoCo | Net difference (AutoLoCo − DDP) |
|---|---|---|---|
| IFEval Strict (all prompt requirements met) | 201/541 (37.15%) | 178/541 (32.90%) | −23 prompts |
| HumanEval (base tests passed) | 50/164 (30.49%) | 45/164 (27.44%) | −5 problems |
| HumanEval+ (base and additional tests passed) | 42/164 (25.61%) | 37/164 (22.56%) | −5 problems |
Q3. Do the workers use all-reduce or a parameter server?
For all-reduce, we use the decentralized peer-to-peer all-reduce in the Hivemind implementation of OpenDiLoCo. All workers contribute equally. Participating peers reduce assigned portions of the update and return the averages. For interval selection, a designated worker coordinates the choice of the next interval. This role deals with scalar statistics and decisions. The model update itself is averaged across the peers.
The update procedure is as follows:
For each interval:
1. On each worker i, in parallel:
w_i = local_AdamW(theta, local_data_i, H) # retain local optimizer state
delta_i = theta - w_i
g = peer_allreduce_mean(delta_i across workers)
2. On every worker, apply the Nesterov outer update with AutoLoCo's outer correction.
3. The leader selects the next H using drift, coherence, and token counts.
4. All workers receive and acknowledge the next H.
Q4. What is the ratio of gradient communication to the total iteration time, how much does DDP overlap, and what would change at 70B?
The ratio of gradient communication time to the total iteration time is shown below.
Table R23. Time spent synchronizing
| Task and workers | Method | Syncs | Training time (h) | Time for synchronization |
|---|---|---|---|---|
| UltraChat 8B, 16 | DiLoCo | 65 | 7.2312 | 58.10% |
| UltraChat 8B, 16 | AutoLoCo | 49 | 6.4958 | 53.39% |
DDP overlap. In our eight workers 8B DDP run, NCCL starts reducing a bucket when its gradients are ready, while backpropagation continues through earlier layers. Across the eight ranks, the average communication span is 3.466 s per measured iteration. A total of 3.243 s overlapped with backward computation. This measurement uses NCCL for multi-GPU communication within a single node. DDP is not suitable for multi-node training in our setting due to its high communication overhead, which is approximately 130 times that of AutoLoCo.
Can AutoLoCo overlap? AutoLoCo could support overlap similar to that in Streaming DiLoCo, allowing workers to continue local training while their pseudo-gradients are being averaged. An initial implementation could retain full-model exchanges. After local steps, each worker would take a snapshot of its pseudo-gradient, keep the outer optimizer coefficients fixed for that round, and resume local training.
Each worker could compute its local contribution to the drift statistic from its snapshot. Once the averaged pseudo-gradient is available, the controller would compute the coherence statistic and select the next synchronization interval. Each worker would then merge the weights produced by the outer update with its current local weights using a predefined rule. This design would combine Streaming DiLoCo–style overlap with AutoLoCo’s approach to reducing synchronization rounds and adjusting the outer update to each interval’s length.
Llama-3.1-70B. Due to the high computational cost of training a 70B model, we will conduct these measurements in future work. Scaling the peak training memory measured in our 8B run by parameter count, we estimate that a 70B replica would need about 8.8 times as much GPU memory, requiring sharding across multiple H20-3e GPUs. The workers will wait about 8.8 times as long for aggregation in each round. Over the same number of rounds, the ratio of training time spent waiting would be about 2.6 times its 8B value
Q5. How does AutoLoCo compare with Local SGD and Streaming DiLoCo?
Local SGD used no momentum, a peak learning rate of 0.01, weight decay of 0.004, and direct parameter averaging every 500 steps. AutoLoCo used local AdamW, adaptive intervals, and a Nesterov outer update. Both settings ran on eight 5090s.
Table R24. Local SGD, DiLoCo, and AutoLoCo on C4 at 50,000 steps.
| Method | Training loss ↓ | Number of syncs ↓ | Payload per sync |
|---|---|---|---|
| Local SGD | 3.4436 | 100 | 1× |
| DiLoCo | 2.8004 | 100 | 1× |
| AutoLoCo | 2.7843 | 73 | 1× |
Table R25. C4 training loss.
| Method | 10K | 20K | 30K | 40K | Final 1K |
|---|---|---|---|---|---|
| Local SGD | 3.7428 | 3.6226 | 3.5289 | 3.4592 | 3.4436 |
| AutoLoCo | 3.1801 | 3.0382 | 2.9252 | 2.8275 | 2.7843 |
Due to time constraints, we were unable to run experiments with Streaming DiLoCo. I think the implementations of Streaming DiLoCo and AutoLoCo do not conflict. In the context of local-update training, they address two separate issues: the number of communication rounds and the time workers spend waiting in each communication step. As discussed in the Can AutoLoCo overlap? section, AutoLoCo can implement communication-computation overlap to mimic Streaming DiLoCo and further reduce communication overhead.
Thank you for your review and valuable feedback!
W1. Where does AutoLoCo help, and how does it fit with other forms of parallelism?
AutoLoCo is not just meant for weak or low-bandwidth networks. It also works across multiple nodes and in setups that rely on local updates. These kinds of training setups are actually quite common. For instance, methods like Local SGD and Local AdamW average model parameters after taking several local steps. Earlier experiments have also tested similar conditions, such as running FedAvg with download speeds under 10 Mbit/s, and testing DiLoCo over 1 Gbit/s connections.
The unit that trains locally can also be a group of GPUs. AutoLoCo treats each worker as a logical training unit that trains a model replica on its own data. Its scheduler determines when workers exchange model updates. Within a worker, the local training system can use pipeline parallelism, FSDP, expert parallelism, or tensor parallelism.
W2. Which models, datasets, and baselines does the evaluation cover?
We compare fixed interval local methods (Local SGD and DiLoCo), feedback interval selection (Adaptive Periodic Averaging), non-feedback schedules (Linear Interval, linear growth, and QSR), and DDP across two Llama-family models like 215M model pretrained on C4 and Llama-3.1-8B fine-tuned on UltraChat-200k and report losses on both datasets and scores on IFEval, IFBench, Multi-IF, MMLU-Pro, MixEval-Hard, BBH, MuSR, MATH, HumanEval, and HumanEval+.
Table R17. C4 pretraining baselines.
Q1. Which GPUs did the workers use, and how were they connected?
Table R18. Hardware and worker placement.
In our 8-worker experiments, we treat each GPU as a separate logical node. Each node maintains its own historical state and communicates independently through Hivemind peer-to-peer connections. In our 16-worker experiments, we use 16 nodes, each serving as one worker with one GPU. The effective throughput measured per-worker ranges from 1.26 to 1.46 Gbit/s. The workers use Hivemind peer-to-peer all-reduce to average the pseudo-gradients across workers. Each worker holds one replica and its own inner optimizer state.
Our scheduling method does not focuses on how inner worker GPU trains its model, but on communication between workers.