acceptodds
Under review as a conference paper at ICLR 2027

AutoLoCo: Communication Efficient Distributed LLM Training via Adaptive Synchronization

Abstract

The pre-training of Large Language Models (LLMs) is increasingly conducted across multiple data centers. As training scales to a larger number of accelerators, the fraction of time spent on computation decreases, while the fraction spent on communication increases. Therefore, frequent synchronization becomes a growing bottleneck. Local update methods reduce this cost by allowing workers to perform several optimizer steps between synchronizations. Most local update methods set the number of local optimizer steps between synchronizations before training and keep this interval fixed throughout the run. However, the best interval can change during the entire train process. If the interval and optimizer are adapted to the current training state, the communication frequency is reduced while maintaining the training performance. In this work, we introduce AutoLoCo, an adaptive training framework to reduce communication in LLM training. It adapts the local interval using scalar training statistics and corrects each outer update. Our method is motivated by two observations: 1) the appropriate local interval varies across training stages, and 2) changing the number of inner steps per interval creates a mismatch with an unchanged outer optimizer, requiring a correction to the outer update. We optimize this mismatch by correction of the outer optimizer for the momentum and the learning rate using the accumulated inner learning rate. Our experiments under communication constraints demonstrate that AutoLoCo reduces communication frequency by 27% relative to DiLoCo while maintaining training performance.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM, Distributed Training, Efficient Communication, Adaptive, Synchronization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_4ZS6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper propose an adaptive framework AutoLoCo for distributed training. Specifically, it reduces the gradient synchronization frequency in data parallelism, with horizon controll that preserves training performance. The technical part is well written, however, the experiments and results presentation need to be improved and need clarification. Additionally, the use case of this work is not clearly defined, i.e., slow networks between GPUs where gradient communication is the bottleneck. With this assumption, "1.26 to 1.46 Gbit/s" GPU-to-GPU connection used in the experiment is not representative for modern GPU clusters (e.g., NVIDIA A100 GPUs with intra-node NVLink and inter-node infiniband/RoCEv1/RoCEv2). Furthermore, the framework claims communication efficiency for distributed LLM training. However, it is designed for data parallelism only, ignoring other communication patterns such as pipeline parallelism, fully-sharded data parallelism, expert parallelism, and tensor parallelism, which are identified as communication bottlenecks by existing work for modern LLM training. Above, I am not convinced about the generality of AutoLoCo's effectiveness, or this work has a narrow use case.

Critical strengths
  1. The technical part (Section 3) is well written and easy to follow.
  2. The implementation details are provided and contribute to the reproducibility of this work.
Critical weaknesses
  1. The scope of this work is not clearly defined. AutoLoCo will be effective only if gradient communication (e.g, Allreduce) is the bottleneck in LLM training, instead of allgather/all-to-all model weights. To demonstrate its effectiveness, the experiment used a small LLM (8B) and an uncommon setting in practical system settings, i.e., 1.26 to 1.46 Gbit/s for inter-worker connections.

  2. The evaluation does not validate the work's effectiveness. Only one model (LLama-3.1-8B and a 215M variant), one dataset (C5), and one baseline (DiLoCo) are used in the experiment.

Questions for authors
  1. Which GPU is used for the workers, and how are GPUs connected? The hardware specifications are important for identifying the scope of this work because the use case of AutoLoCo is when GPU connections are slow. "1.26 to 1.46 Gbit/s" reported in Section 4 is not common for a practical distributed LLM training system.

  2. How many more steps does AutoLoCo need to take to reach the 2.72 loss? Table 5 shows some degradation in downstream tasks with higher loss (IFEval Strict and HumanEval).

  3. What is the communication pattern? Is it all-reduce by treating the workers all the same, or use a parameter server to collect the updates?

  4. What is the ratio of gradient communication to the total iteration time in the experiments? Specifically, what portion of the gradient communication can be overlapped with the bucket communication in DDP, and what will the ratio be when training larger models, e.g., Llama-3.1-70B?

  5. What will be the AutoLoCo performance compared to local SGD and streaming DiLoCo?

3 comments

Thank you for your review and valuable feedback!

W1. Where does AutoLoCo help, and how does it fit with other forms of parallelism?

AutoLoCo is not just meant for weak or low-bandwidth networks. It also works across multiple nodes and in setups that rely on local updates. These kinds of training setups are actually quite common. For instance, methods like Local SGD and Local AdamW average model parameters after taking several local steps. Earlier experiments have also tested similar conditions, such as running FedAvg with download speeds under 10 Mbit/s, and testing DiLoCo over 1 Gbit/s connections.

The unit that trains locally can also be a group of GPUs. AutoLoCo treats each worker as a logical training unit that trains a model replica on its own data. Its scheduler determines when workers exchange model updates. Within a worker, the local training system can use pipeline parallelism, FSDP, expert parallelism, or tensor parallelism.

W2. Which models, datasets, and baselines does the evaluation cover?

We compare fixed interval local methods (Local SGD and DiLoCo), feedback interval selection (Adaptive Periodic Averaging), non-feedback schedules (Linear Interval, linear growth, and QSR), and DDP across two Llama-family models like 215M model pretrained on C4 and Llama-3.1-8B fine-tuned on UltraChat-200k and report losses on both datasets and scores on IFEval, IFBench, Multi-IF, MMLU-Pro, MixEval-Hard, BBH, MuSR, MATH, HumanEval, and HumanEval+.

Table R17. C4 pretraining baselines.

Training method typeMethodTraining loss ↓Number of syncs ↓Payload per sync
Synchronous data parallelismDDP2.722750,0001×
Fixed-interval local trainingLocal SGD (H=500H = 500)3.44361001×
Fixed-interval local trainingDiLoCo2.80041001×
Non-feedback scheduleLinear Interval (model averaging)2.87061001×
Non-feedback scheduleQSR (model averaging)2.8914411×
Non-feedback scheduleLinear growth + outer correction2.7940731×
Non-feedback scheduleQSR + outer correction3.0080731×
Feedback-based interval selectionAdaptive Periodic Averaging (APA)2.8688981×
Feedback-based interval selectionAutoLoCo2.7843731×

Q1. Which GPUs did the workers use, and how were they connected?

Table R18. Hardware and worker placement.

ExperimentWorkersGPUsCommunication between workers
C4, original DiLoCo and AutoLoCo runs88 NVIDIA GeForce RTX 5090Hivemind peer-to-peer all-reduce
C4, four-worker ablation44 NVIDIA GeForce RTX 5090Hivemind peer-to-peer all-reduce
C4, continuation and Local SGD88 NVIDIA GeForce RTX 5090Hivemind peer-to-peer all-reduce
UltraChat, eight workers(DDP)88 NVIDIA H20-3eNCCL
UltraChat, eight workers(DiLoCo/AutoLoCo)88 NVIDIA H20-3eHivemind peer-to-peer all-reduce
UltraChat, sixteen workers1616 NVIDIA H20-3eHivemind peer-to-peer all-reduce

In our 8-worker experiments, we treat each GPU as a separate logical node. Each node maintains its own historical state and communicates independently through Hivemind peer-to-peer connections. In our 16-worker experiments, we use 16 nodes, each serving as one worker with one GPU. The effective throughput measured per-worker ranges from 1.26 to 1.46 Gbit/s. The workers use Hivemind peer-to-peer all-reduce to average the pseudo-gradients across workers. Each worker holds one replica and its own inner optimizer state.

Our scheduling method does not focuses on how inner worker GPU trains its model, but on communication between workers.

Q2. How many extra steps did AutoLoCo need to reach 2.72, and what happened on IFEval Strict and HumanEval?

C4 training loss. We repeated the AutoLoCo training run with more optimization steps. All other training settings were the same as in the original run. The number of steps to reach 2.72 is 109,935.

Table R20. The full AutoLoCo trajectory.

Method60K80K100K109,935
AutoLoCo2.77162.74782.72922.7248

Downstream scores. IFEval Strict counts a prompt as correct only when the response satisfies every requirement, so a single missed instruction causes the whole prompt to fail. On HumanEval, a net difference of five solved problems was present on the base tests. I think that less frequent synchronization is the main reason for the performance gap. DDP synchronized gradients at each of the 6,490 optimizer steps. With AutoLoCo, each worker took consecutive steps using its local data and its own AdamW state, and the workers performed only 49 outer synchronizations over the entire run. These two update schemes lead to different parameter trajectories. Like other local update methods, AutoLoCo shows a performance gap relative to globally synchronized training, although it performs better than the other local update methods we evaluated.

Table R21. Downstream scores (%) from Tables 5 and 6.

MetricDDP, 8 workersDiLoCo, 8 workersAutoLoCo, 8 workersDiLoCo, 16 workersAutoLoCo, 16 workers
IFEval Strict ↑37.1532.7232.9034.0134.38
HumanEval+ pass@1 ↑25.6115.2422.5617.0723.17

Table R22. Performance Gap on Downstream Tasks.

8-worker evaluationDDPAutoLoCoNet difference (AutoLoCo − DDP)
IFEval Strict (all prompt requirements met)201/541 (37.15%)178/541 (32.90%)−23 prompts
HumanEval (base tests passed)50/164 (30.49%)45/164 (27.44%)−5 problems
HumanEval+ (base and additional tests passed)42/164 (25.61%)37/164 (22.56%)−5 problems

Q3. Do the workers use all-reduce or a parameter server?

For all-reduce, we use the decentralized peer-to-peer all-reduce in the Hivemind implementation of OpenDiLoCo. All workers contribute equally. Participating peers reduce assigned portions of the update and return the averages. For interval selection, a designated worker coordinates the choice of the next interval. This role deals with scalar statistics and decisions. The model update itself is averaged across the peers.

The update procedure is as follows:

For each interval:
    1.  On each worker i, in parallel:
            w_i = local_AdamW(theta, local_data_i, H)   # retain local optimizer state
            delta_i = theta - w_i
        g = peer_allreduce_mean(delta_i across workers)
    2.  On every worker, apply the Nesterov outer update with AutoLoCo's outer correction.
    3.  The leader selects the next H using drift, coherence, and token counts.
    4.  All workers receive and acknowledge the next H.

Q4. What is the ratio of gradient communication to the total iteration time, how much does DDP overlap, and what would change at 70B?

The ratio of gradient communication time to the total iteration time is shown below.

Table R23. Time spent synchronizing

Task and workersMethodSyncsTraining time (h)Time for synchronization
UltraChat 8B, 16DiLoCo657.231258.10%
UltraChat 8B, 16AutoLoCo496.495853.39%

DDP overlap. In our eight workers 8B DDP run, NCCL starts reducing a bucket when its gradients are ready, while backpropagation continues through earlier layers. Across the eight ranks, the average communication span is 3.466 s per measured iteration. A total of 3.243 s overlapped with backward computation. This measurement uses NCCL for multi-GPU communication within a single node. DDP is not suitable for multi-node training in our setting due to its high communication overhead, which is approximately 130 times that of AutoLoCo.

Can AutoLoCo overlap? AutoLoCo could support overlap similar to that in Streaming DiLoCo, allowing workers to continue local training while their pseudo-gradients are being averaged. An initial implementation could retain full-model exchanges. After HtH_t local steps, each worker would take a snapshot of its pseudo-gradient, keep the outer optimizer coefficients fixed for that round, and resume local training.

Each worker could compute its local contribution to the drift statistic DtD_t from its snapshot. Once the averaged pseudo-gradient is available, the controller would compute the coherence statistic CtC_t and select the next synchronization interval. Each worker would then merge the weights produced by the outer update with its current local weights using a predefined rule. This design would combine Streaming DiLoCo–style overlap with AutoLoCo’s approach to reducing synchronization rounds and adjusting the outer update to each interval’s length.

Llama-3.1-70B. Due to the high computational cost of training a 70B model, we will conduct these measurements in future work. Scaling the peak training memory measured in our 8B run by parameter count, we estimate that a 70B replica would need about 8.8 times as much GPU memory, requiring sharding across multiple H20-3e GPUs. The workers will wait about 8.8 times as long for aggregation in each round. Over the same number of rounds, the ratio of training time spent waiting would be about 2.6 times its 8B value

Q5. How does AutoLoCo compare with Local SGD and Streaming DiLoCo?

Local SGD used no momentum, a peak learning rate of 0.01, weight decay of 0.004, and direct parameter averaging every 500 steps. AutoLoCo used local AdamW, adaptive intervals, and a Nesterov outer update. Both settings ran on eight 5090s.

Table R24. Local SGD, DiLoCo, and AutoLoCo on C4 at 50,000 steps.

MethodTraining loss ↓Number of syncs ↓Payload per sync
Local SGD3.44361001×
DiLoCo2.80041001×
AutoLoCo2.7843731×

Table R25. C4 training loss.

Method10K20K30K40KFinal 1K
Local SGD3.74283.62263.52893.45923.4436
AutoLoCo3.18013.03822.92522.82752.7843

Due to time constraints, we were unable to run experiments with Streaming DiLoCo. I think the implementations of Streaming DiLoCo and AutoLoCo do not conflict. In the context of local-update training, they address two separate issues: the number of communication rounds and the time workers spend waiting in each communication step. As discussed in the Can AutoLoCo overlap? section, AutoLoCo can implement communication-computation overlap to mimic Streaming DiLoCo and further reduce communication overhead.

Reviewer_Wj9cOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The proposed approach builds upon DiLoCo by allowing it to have an adaptable number of local training steps between synchronizations (rather than fixed). There are other approaches that have done this, the proposed approach both does this using statistics measuring worker displacement and update agreement, but also adjusts pseudo-gradient magnitude, outer momentum and learning rate to account for changing intervals

Critical strengths

I like the fact that the paper thinks about adjusting outer momentum and learning rate due to the changing of H. I also like the use of robust statistics in the controller.

Critical weaknesses

The paper discusses a few adaptive approaches from the literature such as STL-SGD, Adaptive Periodic Averaging, AdaComm, and Dynamic Model Averaging, but only compares to QSR and Linear Interval without explaining why other approaches are not compared to. There should especially be a comparison to feedback-based methods such as Adaptive Periodic Averaging to show that the proposed approach has benefits beyond simply responding to worker disagreement.

The controller (which does the adaptive interval selection procedure) is very heuristical, with no clear indication for whether those choices make sense. It would be good to determine whether unusually high normalized drift or declining coherence actually predicts a loss penalty from extending the interval. I feel like the diagnostic signals used (DtD_t and CtC_t) make sense and so does the use of their robust statistics. However the thresholds are extremely heuristical with no clear intuition on their choice or results on their sensitivity. There are many more complex rules used in the approach, and there is no baseline showing that a simpler approach or even a subset of these choices are inferior.

The design decisions around the outer optimization corrections are reasonably motivated, but not shown to be convincing. First is the update normalization, which is shown to reduce performance in some cases and improve performance in others. Personally I feel like this makes the least sense: this is equivalent to increasing the outer learning rate which has shown to be worse for DiLoCo, and outer steps are not true directions with which to increase. Second is the momentum rule, which makes sense: it ensures the old momentum decays identically regardless of the new length of the outer step, but the key issue is that it does not fix the momentum magnitude (extra reduction of the old momentum to match means the new gradient should be increased so that momentum magnitude is equivalent). The results show that the current version reduces performance, and this is not explored deeper. Finally, the outer learning rate scaling has no clear justification. The steady state makes sense to look at, but why it should be scaled by κt\kappa_t is not clear. Also κmax\kappa_\text{max} is yet another unexplained choice. Performance improves with this addition, and I wonder if it is due to fixing the magnitude issue with the momentum.

Questions for authors

The token aligned interval selection feels both randomly added and not justified. I haven't seen anyone in the literature explicitly care about its impact for these type of approaches, and the paper doesn't give any results on whether this helped or not. Can the authors give ablations for this, and/or report how much token density varies across synchronization intervals and how substantially the mapper changes the executed horizons? Is there any specific reason why this selection felt neccessary?

5 comments

Thank you for highlighting these important questions. We address them separately.

W1. Comparison with Adaptive Periodic Averaging

Our original comparison covered two ways to schedule communication. Linear Interval increases the interval on a fixed schedule, and QSR ties it to the inner learning rate. This left out a method that responds to disagreement between workers. We have added Adaptive Periodic Averaging (APA) [1] on the 215M C4 model. Both methods use eight workers, local AdamW, 128 examples per worker per step, and 50,000 optimizer steps. APA uses the variance of worker endpoints to adjust the interval and averages the model parameters directly. The actual full interval range for APA is 500–548 steps.

Table R8. Feedback methods on C4 with eight workers.

MethodTraining loss ↓Number of syncs ↓Payload per syncValidation loss ↓
Adaptive Periodic Averaging2.8688981×2.8680
DiLoCo, momentum 0.92.80041001×2.8020
AutoLoCo2.7843731×2.7842

Table R9. Training loss over time.

Method10K20K30K40KFinal 1K
Adaptive Periodic Averaging3.27353.08672.97132.89302.8688
DiLoCo, momentum 0.93.16143.03152.92372.83572.8004
AutoLoCo3.18013.03822.92522.82752.7843

AutoLoCo used 25 fewer synchronizations than APA (25.5%) and finished with a training loss lower by 0.0845. Its training loss was also lower at each milestone in Table R9. This comparison measures the performance of each feedback method with its own outer update.

W2a. Do drift and coherence predict the cost of extending an interval?

Let Δt(i)\Delta_t^{(i)} denote the shared starting parameters minus the final parameters of worker ii in interval tt. Let Δˉt\bar{\Delta}_t be the mean of these displacements across the NN workers. Section 3.1 defines drift as the mean squared distance from the shared starting point to the workers' endpoints,

Dt=1N∑i=1N∥Δt(i)∥22.D_t = \frac{1}{N} \sum_{i=1}^{N} \lVert \Delta_t^{(i)} \rVert_2^2.

The coherence statistic is

Ct=N∥Δˉt∥22Dt+ϵ.C_t = \frac{N \lVert \bar{\Delta}_t \rVert_2^2}{D_t + \epsilon}.

The ratio Ct/NC_t/N measures the fraction of displacement energy retained after averaging, with ϵ\epsilon stabilizing the denominator. The mean squared distance of the workers' endpoints from their mean is

= D_t - \lVert \bar{\Delta}_t \rVert_2^2.$$ Workers can therefore move far from their shared starting point and still agree on the direction of the update. The controller accounts for the drift expected from the sum of the inner learning rates, then measures excess drift and a drop in coherence relative to recent intervals at the accepted $H$. We examined the first extension in two runs with eight workers and base outer momentum 0.7. At checkpoint step 3,500, both methods had just completed an interval of 500 steps. AutoLoCo then tried 550 steps. It ran five intervals of 550 steps before choosing 600 at step 6,250. DiLoCo kept $H=500$ throughout. Over steps 3,501 to 6,250, mean training loss was **3.393614** for AutoLoCo and **3.393758** for DiLoCo. Over steps 3,501–4,500 the means were 3.467209 for AutoLoCo and 3.468507 for DiLoCo. Over steps 4,501–5,500 they were 3.373228 and 3.378047. DiLoCo had the lower loss over the remaining 750 steps, at 3.315039 versus 3.322669 for AutoLoCo. The difference over the entire window was 0.000144. Under the same number of training steps, AutoLoCo extends the communication intervals without degrading training performance, suggesting that drift and coherence have some predictive value in determining when communication can be reduced.

W2b. Choice of thresholds and sensitivity

The three thresholds serve different decisions. A score below 1.5 supports trying a longer interval. At 2.5 the controller rejects a candidate, and 4.0 marks a severe departure. The units come from the current run. We center the residuals at their historical median and measure their spread with the median absolute deviation. This gives the controller a stricter criterion for proposing growth than for keeping an interval it has already tried. We varied all three thresholds together by 20% in either direction on C4 with four workers and seed 42. Both new runs ended after six consecutive stable monitoring intervals at H=750H=750, at step 23,500. We compare the original run at the same step. The effective batch was 128 per worker throughout.

Table R10. Results at the common endpoint of 23,500 steps.

MethodTraining loss ↓Number of syncs ↓Payload per sync
AutoLoCo (1.2, 2.0, 3.2)3.0331371×
AutoLoCo (1.5, 2.5, 4.0)3.0326371×
AutoLoCo (1.8, 3.0, 4.8)3.0325371×

Table R11. Training loss for the threshold settings.

Method10K20K23.5K
AutoLoCo (1.2, 2.0, 3.2)3.20563.08523.0331
AutoLoCo (1.5, 2.5, 4.0)3.20583.08533.0326
AutoLoCo (1.8, 3.0, 4.8)3.20643.08523.0325

All three settings proposed the five increases at the same steps. They selected 750 at step 16,000, accepted it after the first trial at step 16,750, and completed 37 synchronizations by step 23,500. A stricter growth cutoff can delay an extension, and a more permissive cutoff can advance it. For this seed and training window, shifting all three thresholds together by 20% in either direction reproduced the default synchronization schedule with comparable training loss. AutoLoCo showed low sensitivity to the tested thresholds in this experiment.

W2c. Which controller rules determined the observed increases?

For the controlling process, three steps determined the increases in workers.

  1. First, the controller estimated the usual drift and coherence at the accepted HH.
  2. It then proposed a slightly longer interval.
  3. Finally, it ran that candidate for a full interval and used the new observations to decide whether to accept it. After acceptance, it estimated a new reference at the new HH.

During implementation, we included several additional controller state-transition rules to handle edge cases, anticipating more complex and larger-scale training scenarios in the future. Based on our monitoring of the actual 215M model and LLaMA-3.1-8B training runs, the following rules of 2,3,5 were triggered in practice, and 1, 4 were minimal safeguards for edge cases. We will provide a more concise description in a future revision.

Table R12. Core rules in controller

NumberConditionNext horizon
1Statistic broken, or z≥4.0z \ge 4.0 twiceReset to Hmin⁡H_{\min}
2HaH_a, z<1.5z < 1.5, and a larger admissible HcH_c existsIncrease to HcH_c for one trial
3Trial at HcH_c, z<2.5z < 2.5Accept and keep HcH_c
4Trial at HcH_c, z≥2.5z \ge 2.5Reduce to HaH_a
5Otherwise at HaH_aKeep HaH_a

W3a. Does pseudo-gradient normalization increase the outer learning rate?

No. In AutoLoCo, enabling normalization scales the current pseudo-gradient by 1/ρt1/\rho_t. When ρt>1\rho_t>1, this reduces the effective step size on the new pseudo-gradient. The incoming momentum retains the same coefficient in the current parameter update.

Let η\eta and μ\mu denote the outer learning rate and momentum coefficient for the current interval. Given the same raw pseudo-gradient Δˉt\bar{\Delta}_{t} and incoming momentum vt−1v_{t-1}, the three choices give the following coefficients in the Nesterov parameter update θt−θt+1\theta_{t} - \theta_{t+1}.

OperationCoefficient on the new pseudo-gradient Δˉt\bar{\Delta}_tCoefficient on the incoming momentum vt−1v_{t-1}
Normalization offη(1+μ)\eta(1+\mu)ημ2\eta\mu^2
Normalization onη(1+μ)/ρt\eta(1+\mu)/\rho_tημ2\eta\mu^2
Normalization off, outer learning rate reduced to η/ρt\eta/\rho_tη(1+μ)/ρt\eta(1+\mu)/\rho_tημ2/ρt\eta\mu^2/\rho_t

The last two rows give the new pseudo-gradient exactly the same weight in the current parameter update. With normalization, the momentum buffer receives Δˉt/ρt\bar{\Delta}_t/\rho_t, so this interval also carries less weight in subsequent updates. When we reduce only the outer learning rate, the buffer receives the full Δˉt\bar{\Delta}_t, while the smaller learning rate scales the contributions of both the new pseudo-gradient and the incoming momentum to the current parameter update.

The training task matters because the two settings differ in how much data each worker sees per inner step(sampling effects). In the 16-worker UltraChat setup, each worker computes a local update using only two conversations, so the mix of supervision can differ substantially across workers at the same step. In C4, each worker processes 128 documents per inner step. Under representative sampling, this larger batch can give a more stable estimate of the overall data mixture.

In C4, we expect the direction accumulated over a longer interval to be useful on other documents as well. We disable normalization so that the full displacement accumulated across those documents contributes to the outer update.

In UltraChat, an interval that happens to contain many conversations of a similar type may pull the local update toward that type of supervision. Normalization gives this interval less weight in both the current parameter update and the momentum carried into the next interval. Directions that recur across intervals with different samples can already be represented in the incoming momentum. We therefore enable normalization in UltraChat to limit how strongly one interval’s sample mix can change the direction of the outer update.

Reviewer_SmHgOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

AutoLoCo combines adaptive synchronization intervals with corrections to the DiLoCo outer optimizer. Its controller uses local displacement and aggregation coherence to select a token-equivalent horizon. The outer correction uses accumulated inner learning rates to adjust momentum and the outer learning rate, with pseudo-gradient normalization enabled for SFT but disabled for the reported C4 recipe. Evaluation covers 215M C4 pretraining and Llama-3.1-8B full-parameter UltraChat fine-tuning, including a physical-network experiment with 16 workers.

Critical strengths
  1. The interaction between synchronization timing and the outer optimizer is a useful contribution. A longer interval changes both the local displacement and the amount of inner optimization represented by one outer update. Section 3.2 addresses this by comparing accumulated learning rates from the same scheduler position. Table 7 supports coordinating the corrections: adaptive intervals alone reduce synchronization from 100 to 73 rounds but slightly worsen training loss (2.8251 to 2.8310); adding only momentum adjustment to the adaptive-interval variant gives 2.8484, whereas the complete correction gives 2.8116. Appendix E.4 explicitly matches DiLoCo's base momentum to 0.7 in this ablation. This supports using the interval and optimizer corrections together.

  2. The paper measures practical benefits in addition to synchronization counts. In Table 1, AutoLoCo uses 73 rather than 100 synchronization rounds and improves C4 validation loss from 2.8020 to 2.7842. More importantly for the systems motivation, the 16-worker physical-network experiment in Table 4 reduces wall-clock time from 7.2312 to 6.4958 hours, approximately 10.2%, while improving test NLL from 0.9016 to 0.8772. This supports an end-to-end benefit in the evaluated setting. The distinction between the simulated eight-worker experiments and this physical-network experiment in Appendix E.1 is helpful; the logical payload reduction should remain distinct from the smaller measured runtime reduction.

Critical weaknesses
  1. The additional value of online horizon selection over a simpler horizon policy is not yet sufficiently isolated. The Linear Interval and QSR comparisons in Table 1 are useful, but Section 4.1 and Appendix E.2 state that they use direct model averaging, whereas AutoLoCo uses a corrected Nesterov outer update. These comparisons therefore change both the scheduling policy and the outer update. Appendix F.2 also reports a trajectory with 45 intervals at H=750, accounting for 33,750 of the 50,000 training steps. The earlier growth phase could still be important, motivating a comparison with a longer fixed horizon or a simple predetermined growth schedule under the same outer correction. Such a control would clarify whether online feedback contributes beyond choosing a suitable trajectory and optimizer recipe.

  2. Attribution and repeatability of the main gains need a more complete account. The matched-momentum four-worker ablation is a strength, but it does not fully resolve the main-scale comparisons: Appendix E.3 specifies DiLoCo momentum 0.9 for SFT, while Table 12 lists AutoLoCo's base momentum as 0.7. For C4, Table 12 lists 0.7/0.9 without mapping these values to individual runs. I would like this mapping and the tuning procedure made explicit. Table 11 and Appendix E.4 report seed 42, with no variation across independent training runs reported. Independent runs would clarify the repeatability of the smaller quality differences and the measured runtime gain. Normalization is also recipe-dependent in Tables 8-9, so the configuration summary and Algorithm 1 should make its optional status explicit.

My weak-rejection recommendation reflects the remaining uncertainty about the incremental benefit of online feedback over simpler policies with comparable outer optimization. A matched policy comparison and clearer evidence of repeatability would make my assessment more positive.

Questions for authors
  1. With the same outer update/correction, base optimizer settings, and training budget, how does online selection compare with a longer fixed horizon and a simple growth schedule? Please distinguish this control from the direct-averaging Linear Interval and QSR comparisons already reported.

  2. Which C4 runs use each momentum value in Table 12, and how were the baseline and AutoLoCo settings tuned? Do matched-base-momentum comparisons at the main scales support the same conclusions as Table 7? Please also make the normalization switch explicit in the configuration table and pseudocode.

  3. Can you report variation across independent training runs for the main quality gains and, where feasible, repeated measurements for the 16-worker runtime comparison? Any existing repeated-run evidence would help distinguish stable improvements from small run-to-run differences.

3 comments

Thank you for the review. We discuss the horizon policy controls, optimizer settings, and evidence from independent runs below. We have also added new experimental results and updated the paper following your suggestions.

W1 and Q1

We added Linear growth + Outer correction and QSR + Outer correction. These controls use 8 workers, the 215M model, C4, and 50,000 optimizer steps. They retain the inner AdamW recipe, base outer learning rate 0.7, base momentum 0.7, and the momentum and learning rate corrections in Section 3.2. As in the C4 AutoLoCo, pseudo-gradient normalization is disabled. Linear Interval is a predetermined growth policy. Our added control starts at H = 500 and increases the horizon by 50 up to 750, following a schedule fixed before training. It shares AutoLoCo's growth increment and upper bound but does not use the observed drift or coherence. This schedule has the same total number of synchronizations as AutoLoCo. QSR + Outer correction retains the inverse square learning rate rule for choosing the horizon and applies the same outer corrections as AutoLoCo.

For the longer fixed horizon, We evaluate H = 700 as a nearby fixed interval. It uses the outer correction update with outer learning rate and momentum both set to 0.7. In Appendix D.2, we described that AutoLoCo recovers fixed-horizon DiLoCo when the selected horizon and outer optimizer remain at their base settings. So for fixed horizon, the outer correction is no different from the original update method used in DiLoCo.

Table R1. Horizon policy comparison on C4 with 8 workers.

MethodTraining loss ↓Number of syncs ↓Payload per syncValidation loss ↓
Linear growth + Outer correction2.7940731×2.7937
QSR + Outer correction3.0080731×3.0076
Fixed H = 7002.8107721×2.8104
AutoLoCo2.7843731×2.7842

Table R2. Training loss over time for the horizon policies.

Method10K20K30K40KFinal 1K
Linear growth + Outer correction3.20483.04772.93502.83092.7940
QSR + Outer correction3.18543.07913.10313.05013.0080
Fixed H = 7003.19873.03532.93192.84382.8107
AutoLoCo3.18013.03822.92522.82752.7843

W2 and Q2

In the main experiments, we use the momentum values that are closer to those used in the original methods and achieve the best results. In the ablation studies, we use the same momentum value for both DiLoCo and AutoLoCo to minimize differences introduced by this variable and ensure a fair comparison. Table R3 clarifies the momentum 0.7/0.9 in experiments.

Table R3. Base outer optimizer settings.

MethodBase outer momentumBase outer LR
DiLoCo (main C4 and UltraChat comparisons)0.90.7
DiLoCo (Table R3 and C4 ablation)0.70.7
AutoLoCo (C4 and UltraChat)0.70.7
Linear growth + Outer correction0.70.7
QSR + Outer correction0.70.7
Fixed H = 7000.70.7
Adaptive intervals only (C4 ablation)0.70.7
Adaptive intervals + momentum correction (C4 ablation)0.70.7

Our DiLoCo setting in main experiments follows the original study [1], which compared outer optimizers and their hyperparameters and adopted Nesterov with learning rate 0.7 and momentum 0.9. OpenDiLoCo [2] uses the same values in its replication and model scaling experiments. We also compared momentum 0.7 and 0.9 in our eight worker C4 setup. Momentum 0.9 gives a final training loss of 2.8004, compared with 2.8041 for 0.7, supporting its use in the main DiLoCo result. Tables R4 and R5 compare the momentum settings using 8 workers and 50,000 optimizer steps.

Table R4. C4 results with explicit momentum settings, using 8 workers.

MethodTraining loss ↓Number of syncs ↓Payload per sync
DiLoCo, momentum 0.92.80041001×
DiLoCo, momentum 0.72.80411001×
AutoLoCo, base momentum 0.72.7843731×

Table R5. Training loss over time for the momentum comparison.

Method10K20K30K40KFinal 1K
DiLoCo, momentum 0.93.16143.03152.92372.83572.8004
DiLoCo, momentum 0.73.18503.02492.92022.83602.8041
AutoLoCo, base momentum 0.73.18013.03822.92522.82752.7843

For the optional status in normalization, we think this can be addressed from the perspective of the data and task. The explanation begins with sampling effects. In 16 worker UltraChat setup, each worker computes a local update using only two conversations, so the composition of supervision can differ substantially across workers at the same step. C4 combines 128 documents per worker per step, providing a more stable estimate of the overall mixture distribution under appropriate sampling. Therefore, normalization can be disabled when a local interval adequately represents the target mixture distribution and extending the interval mainly adds new supervision. On the contrary, it can be enabled when a local interval is dominated by a small number of conversations, tasks, or response patterns.In this situation, local sampling variability is high, and extending the interval continues to accumulate these local biases.

For Algorithm 1, we modified it by adding:

10: if bnorm=1b_{\mathrm{norm}}=1 then
11: Δ~t←Δˉtρt\widetilde{\Delta}_t \gets \frac{\bar{\Delta}_t}{\rho_t}
12: else
13: Δ~t←Δˉt\widetilde{\Delta}_t \gets \bar{\Delta}_t
14: end if

The momentum and learning rate corrections are independent of the normalization switch. As specified in Appendix B.4, changing the momentum coefficient does not rescale the stored momentum buffer.

Q3

We completed two additional four workers AutoLoCo training runs with seeds 100 and 200. All three use the same settings as the seed 42 run in C4 ablation. Across three AutoLoCo runs, the Final 1K training loss is 2.8114 ± 0.0013.

Table R6. Training loss and synchronization counts across the C4 seed runs.

Method10K20K30K40KFinal 1K ↓Number of syncs ↓Payload per sync
DiLoCo, seed 423.20703.04872.94662.85592.82511001×
AutoLoCo, seed 423.20583.08532.96322.85562.8116731×
AutoLoCo, seed 1003.20533.08662.96282.85802.8126731×
AutoLoCo, seed 2003.20513.08292.96142.85512.8100731×

For the runtime comparison, we repeat two AutoLoCo runs on the 16 workers setup. All three system runs train Llama-3.1-8B on UltraChat for 6,490 optimizer steps with global batch 32 and BF16 pseudo-gradient communication. Runtime is measured on worker 0 from training-ready to completion of the final optimizer step, including synchronization and normal gaps between steps.

Table R7. Runtime measurements with 16 workers on UltraChat.

Method / runOptimizer stepsNumber of syncs ↓Training time (hours) ↓Time reduction relative to DiLoCo
DiLoCo6,490657.2312-
AutoLoCo6,490496.495810.17%
AutoLoCo6,490496.343712.27%

References

[1] Douillard et al. DiLoCo: Distributed Low-Communication Training of Language Models. Section 3.1, Outer Optimizers; Table 5.

[2] Jaghouar et al. OpenDiLoCo: An Open-Source Framework for Globally Distributed Low-Communication Training. Replication Experiment Setup; Appendix A.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.