acceptodds
Under review as a conference paper at ICLR 2027

The Stabilizing Effect of Gossip in Decentralized Language Model Pre-Training

Abstract

As language-model training scales beyond a single connected cluster, every-step global synchronization becomes increasingly expensive. Gossip-based decentralized training replaces global all-reduce with local updates and peer-to-peer parameter averaging. We investigate how this change affects training dynamics, beyond its communication benefits. Across the reported model families and optimizer settings, gossip-based decentralized training maintains comparatively stable validation-loss trajectories in large-learning-rate regimes where centralized training becomes highly oscillatory or moves toward high loss. To understand this phenomenon, we compare the training dynamics of decentralized and centralized SGD in deep linear networks. We prove that gossip SGD can achieve strictly lower expected full-Hessian sharpness at the worker-averaged iterate than centralized SGD. Our analysis further establishes conditions under which gossip SGD improves stability to directional parameter perturbations at large learning rates, quantified by reduced multistep sensitivity. This provides a theoretical counterpart to the stability advantage observed in our language-model experiments.

Primary area:
optimization
Keywords:
Decentralized learning, Optimization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_pMB6OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper investigates how gossip communication affects optimization stability in language-model pre-training. Across several model families and optimizers, gossip-averaged models exhibit smoother validation-loss trajectories at aggressive learning rates. A deep-linear-network analysis establishes conditions for lower expected full-Hessian sharpness at the worker-averaged iterate, a mixing-dependent drift toward lower sharpness, and reduced finite-window sensitivity along a specified perturbation direction.

Critical strengths

The empirical observation is interesting and extends across AdamW, SGDM, Muon, several Transformer families, and models up to 1.5B parameters. The SGDM Local SGD comparison and three-seed reporting provide useful checks. A reproducible stability advantage in this setting would be valuable beyond the usual communication-efficiency motivation for gossip.

The theory provides a substantive, carefully delimited contribution. It analyzes the actual worker-averaged iterate and full Hessian, rather than only a surrogate curvature quantity, and separates sharpness reduction from directional perturbation stability. The explicit counterexamples and limitations make the sufficient mechanism easier to assess and prevent an unwarranted universal interpretation.

Critical weaknesses

The empirical attribution to gossip needs a more tightly matched control. Appendix C.1.3 selects H=5 for AdamW from four local-update horizons, while the reported Local SGD control uses SGDM with H=1. That control does not determine whether the main AdamW advantage requires pairwise gossip or also arises from global model averaging every five local updates. In addition, Appendix C.1.2 specifies clipping at norm 1.0 without making its order relative to gradient reduction explicit, and optimizer-state handling after model averaging is unclear. For paired minibatches, common initial momentum states, and no difference in nonlinear clipping, H=1 global parameter averaging under linear SGDM should reproduce the centralized mean update. The paper should clarify whether these conditions hold and whether the Figure 3 separation reflects protocol differences or sampling variability. These details matter directly to what phenomenon has been isolated.

The theory-to-Transformer connection remains suggestive. The theory uses a single prediction direction, fixed mixing, local small-noise conditions, and plain SGD, whereas the language-model runs use random pairings and momentum or adaptive optimization. The manuscript acknowledges these limits. However, the reported raw blockwise curvature does not show that the proved directional mechanism drives the observed stability. Appendix C.1.4 also leaves the curvature sample size, sharing rule, and power-iteration convergence settings unspecified, limiting the diagnostic’s reproducibility. This limits the explanatory claim and prevents a stronger recommendation.

Questions for authors

Could the authors report AdamW Local SGD at H=5 alongside gossip at H=5, and specify the exact order of reduction, clipping, optimizer-state updates, and parameter/state synchronization? Were minibatches paired and initial momentum states matched, and does the Figure 3 separation reflect different update operations or sampling variability?

What sample sets and power-iteration settings were used for the curvature measurements, and does the qualitative ordering persist with a fixed shared batch and a sufficiently converged estimate?

1 comment
AuthorsOpenReview ↗

Response to Reviewer pMB6

Thank you for your support and for highlighting the full-Hessian analysis at the worker-averaged iterate. We appreciate your suggestion to include a matched Local SGD control.

Q1 (Local SGD at H = 5) Could the authors report AdamW Local SGD at H=5 alongside gossip at H=5 …?

A1. Thanks for suggesting this control. We ran a Local SGD-style AdamW control at H = 5, with global averaging every five local steps. We matched the gossip recipe in Figure 2(b) (124M model, peak LR 0.05, batch size, warmup, and local optimizer-state handling). The averaged models had stable validation curves at 25-step checkpoints through step 5,000, reaching 4.18 (Local SGD) and 4.24 (gossip). Each curve shows one run. The new control used a newer runtime. Matched validation curves

This result is consistent with our theory. Theorem F.8 (Appendix F.6) covers the corresponding SGD schedule. Worker deviations affect the mean update between rounds and can accelerate motion toward lower sharpness in the scalar small-noise model. This broadens the empirical scope to Local SGD at the tested setting.

Q2 (Update order and pairing) … specify the exact order of reduction, clipping, optimizer-state updates, and parameter/state synchronization? Were minibatches paired and initial momentum states matched, and does the Figure 3 separation reflect different update operations or sampling variability?

A2. As in standard data parallelism, Centralized training (DDP) averages accumulated gradients before whole-model norm clipping at 1.0 and the optimizer step. Local SGD and Decentralized training (gossip) clip accumulated gradients locally at 1.0 and then step. At communication rounds, they average parameters globally or within pairs, respectively. Momentum buffers and AdamW moments stay local. Section 4.2 and Appendix C.1.3 give the update order.

Clipping before the local step is established practice. Representative examples are DiLoCo [1] (§3) for local updates and DECOR [2] (Algorithm 1) for gossip.

The Figure 3 runs use the same deterministic rank-wise minibatch schedule and start with zero momentum history. Each protocol has three repeats at nominal seed 41 with SGDM momentum 0.9. At H = 1, clipping the averaged gradient and averaging locally clipped gradients define different update rules even with paired batches. Local SGD and gossip share the local update rule and differ in how they average parameters. At peak LR 1.5, their validation curves separate in all three repeats, with mean losses of 34.00 for Local SGD and 3.96 for gossip at step 5,000.

Q3 (Curvature batches and convergence) What sample sets and power-iteration settings were used for the curvature measurements, and does the qualitative ordering persist with a fixed shared batch and a sufficiently converged estimate?

A3. Each checkpoint uses 128 training sequences of length 1,024 (16 per worker). The batch stays fixed across all blocks and iterations within a measurement and advances between checkpoints. We use up to five warm-started iterations per block, following the exact-Hessian comparison in Ramasinghe et al. [3] (§4.2, Figure 2). With this protocol, Figure 3(c) shows smaller maximum absolute blockwise Rayleigh values for gossip at peak LR 1.5.

Q4 (Direct measurement of directional perturbation response) However, the reported raw blockwise curvature does not show that the proved directional mechanism drives the observed stability.

A4. Following your suggestion, we have added a directional perturbation experiment in a 124M Transformer trained with AdamW (Figure 8 and Appendix H.5). We inject perturbations along a fixed AdamW update direction and replay the next 20 updates. At peak LR 0.08, gossip has a smaller squared response at every replay step across four checkpoints and two finite amplitudes. Its 20-step cumulative response is 6.4–40.9% of the Centralized response. Both amplify the perturbation by step 20. Figure 8 and replay protocol

Our appendix theory covers more general SGD settings.

  • Prediction and communication. Theorems F.2 and F.8 cover vector predictions and time-varying Markov mixing, respectively.
  • Directional response. Theorem G.1 covers smooth nonlinear scalar predictors under its local geometry and history conditions.

References

Reviewer_WQz1OpenReview ↗
Rating
2(−1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether gossip-based decentralized training enhances the stability of language model training, beyond its known communication benefits. The authors observe that, at large learning rates, gossip training yields more stable validation-loss trajectories than centralized training across various Transformer architectures and optimizers. To explain this phenomenon, they employ deep linear networks for theoretical analysis. The paper establishes a finite-step reduction in expected full-Hessian sharpness, characterizes how the mixing spectrum influences the drift toward lower-sharpness solutions, and derives conditions for reduced sensitivity to directional parameter perturbations.

Critical strengths
  • The empirical findings are interesting and significant. The observed stability advantage is robust across different Transformer architectures, model sizes, and optimizers (not limited to AdamW). Notably, the inclusion of an H=1H=1 global parameter-averaging control in the SGDM experiments is a valuable addition, as it helps isolate whether the stability gains stem from the averaging mechanism itself or from the specific dynamics of gossip. This provides a solid empirical foundation for viewing gossip as an optimization mechanism, not merely a communication strategy.

  • The theoretical analysis offers more than just an intuitive explanation. The finite-step sharpness result applies to deep linear networks with arbitrary finite hidden widths, and the slow-time analysis explicitly links the mixing spectrum to the drift dynamics. I also appreciate the authors’ effort to distinguish these results from the more restricted perturbation-sensitivity guarantee, including the provision of counterexamples that delineate the boundaries of the theory’s applicability.

Critical weaknesses
  • Several details regarding the training protocol are crucial for interpreting the stability gains but remain unclear. Appendix C.1.2 specifies gradient clipping at norm 1.0 but does not clarify whether clipping is applied before or after cross-worker aggregation, nor how optimizer states are handled during averaging. These choices significantly impact effective updates at large learning rates. Specifically, under matched minibatches and standard linear momentum, global parameter averaging with H=1H=1 should theoretically recover the centralized SGDM parameter recursion unless other operations intervene. The observed differences between these controls therefore require clarification. Furthermore, while the main AdamW gossip experiments use H=5H=5, a corresponding global-averaging control with the same horizon is not reported. This omission leaves uncertainty about how much of the improvement is attributable to persistent worker disagreement versus differences in the update protocol.

  • The connection between the theoretical results and the language model experiments remains indirect. The theory relies on near-fit, small-noise dynamics in scalar-input/output deep linear networks with fixed mixing, whereas the Transformer experiments employ different optimizers and randomized pairwise gossip. While the reported blockwise Rayleigh diagnostics are consistent with reduced curvature, they are not verified estimates of the largest positive full-Hessian eigenvalue. More critically, the experiments do not directly test the directional perturbation response predicted by the theory. Consequently, while the empirical phenomenon is convincing within the tested settings, the proposed theoretical explanation remains a plausible mechanism rather than an established causal link.

Questions for authors
  1. Could you clarify the exact order of operations for gradient clipping and aggregation in each protocol? Specifically, are momentum or AdamW states averaged across workers or kept local? In particular, what factors account for the divergent SGDM trajectories observed between DDP and the H=1H=1 global-averaging control?

  2. Are results available for AdamW with global parameter averaging at H=5H=5, using the same local-update and optimizer-state settings as the gossip baseline?

  3. How many power iterations and curvature samples are used at each checkpoint? Have you verified the convergence of the Rayleigh estimates, or compared them with estimates targeting the largest positive block-Hessian eigenvalue?

1 comment
AuthorsOpenReview ↗

Response to Reviewer WQz1

Thank you for your support and for highlighting the stability results across models and optimizers. We appreciate your suggestion to test the directional response directly.

Q1 (Clipping and optimizer state) Could you clarify the exact order of operations for gradient clipping and aggregation in each protocol? Specifically, are momentum or AdamW states averaged across workers or kept local? In particular, what factors account for the divergent SGDM trajectories observed between DDP and the H = 1 global-averaging control?

A1. Thanks for pointing this out. As in standard data parallelism, Centralized training (DDP) averages accumulated gradients across workers before clipping and stepping. Local SGD and Decentralized training (gossip) clip accumulated gradients locally before stepping, then average parameters globally or within pairs at communication rounds. All use whole-model norm clipping at 1.0. Momentum buffers and AdamW moments stay local across averaging rounds.

Clipping before the local step is established practice. Representative examples are DiLoCo [1] (§3) for local updates and DECOR [2] (Algorithm 1) for gossip.

With matched batches and initialization, unclipped H = 1 linear SGDM gives the same mean update in Centralized training and Local SGD. In our protocols, the clipping order changes the update rule even with the same minibatches. For worker gradients g_i, clip(mean(g_i)) ≠ mean(clip(g_i)) in general. We have clarified the update order and state handling in §4.2 and Appendix C.1.3.

Q2 (Global parameter averaging at H = 5) Are results available for AdamW with global parameter averaging at H = 5, using the same local-update and optimizer-state settings as the gossip baseline?

A2. Thanks for suggesting this control. We added global parameter averaging at H = 5 with AdamW, matching the archived gossip run in Figure 2(b) on model (124M GPT-2-style), peak LR (0.05), batch geometry, warmup, and local optimizer-state handling. Both validation curves stayed stable through step 5,000, reaching 4.18 (global averaging) and 4.24 (gossip) on the averaged models. Each curve shows one run. The new control used a newer runtime. Matched validation curves

This result is consistent with our theory. For SGD, every five-step mixing product equals the global averaging matrix, meeting the contraction condition in Theorem F.8 (Appendix F.6). Worker deviations between rounds affect the mean update (Proposition D.1) and can accelerate motion toward lower sharpness in the scalar small-noise model. This broadens the empirical finding to periodic global averaging at this rate.

Q3 (Curvature measurement and validation) How many power iterations and curvature samples are used at each checkpoint? Have you verified the convergence of the Rayleigh estimates, or compared them with estimates targeting the largest positive block-Hessian eigenvalue?

A3. Following the exact-Hessian comparison in Ramasinghe et al. [3] (§4.2, Figure 2), we use up to five power iterations per block, warm-started from the previous checkpoint. At each checkpoint, we share 128 training sequences of length 1,024 (16 per worker) across blocks and iterations. The loader advances between checkpoints. We target dominant-magnitude block curvature and report the maximum absolute Rayleigh quotient across blocks in Figure 3.

Q4 (Direct test of perturbation response) More critically, the experiments do not directly test the directional perturbation response predicted by the theory.

A4. Following your suggestion, we ran a new perturbation experiment on a 124M AdamW Transformer. We inject along a fixed AdamW update direction and replay with matched minibatches (Figure 8, Appendix H.5). At peak LR 0.08, gossip has a smaller squared response at every replay step across four checkpoints and two finite amplitudes. Its 20-step cumulative response is 6.4–40.9% of the Centralized response. Both amplify the perturbation by step 20. Figure 8 and replay protocol

Our appendix theory covers more general SGD settings.

  • Communication. Theorem F.8 covers Markov mixing under graph-product contraction.
  • Directional response. Theorem G.1 covers smooth nonlinear scalar predictors under its local geometry and history conditions.

References

Reviewer_vp8POpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper compares gossip-based decentralized learning and centralized learning in terms of Hessian sharpness and training stability under large learning rates. Motivated by empirical observations from experiments on several language models, the authors further provide theoretical insights into this phenomenon using simpler deep linear neural networks.

Critical strengths

The paper compares decentralized and centralized learning from a different perspective than what is typically explored in the literature. Specifically, it examines the two approaches in terms of training-loss stability, which is an interesting and valuable aspect to investigate.

Critical weaknesses

1- Insufficient motivation: The paper is mainly motivated by the improved stability of decentralized training at high learning rates. However, the practical significance is unclear, as Figure 2 shows that both centralized and decentralized training achieve comparable or better validation loss at small learning rate settings.

2- Rayleigh diagnostic: In Figure 3, for (\eta=1.5) and (t=1000)–(2000), the validation loss and Rayleigh quotient show different trends. This raises questions about how reliably the blockwise Rayleigh diagnostic represents the largest eigenvalue of the full Hessian.

3- Restrictive theoretical assumptions: The theory relies on deep linear networks, scalar inputs, squared loss, and small-noise conditions. These assumptions are substantially different from the Transformer training setting that motivates the paper, making the connection between theory and experiments unclear.

4- Insufficient explanation: Several aspects need further clarification. For example, what is the difference between Local SGD with (H=1) and centralized training? Additionally, the theorem statements are difficult to follow, with limited discussion of their intuition and implications.

5- Confusing notation: The mathematical notation is sometimes inconsistent or confusing. For example, both (N) and (T) are used to represent training horizons in different sections.

Questions for authors

Please check weaknesses.

1 comment
AuthorsOpenReview ↗

Response to Reviewer vp8P

Thank you for your detailed feedback. We appreciate your interest and your questions about the practical motivation.

Q1 (Why test large learning rates?) … the practical significance is unclear, as Figure 2 shows that both centralized and decentralized training achieve comparable or better validation loss at small learning rate settings.

A1. Thanks for pointing this out. We show that gossip keeps training stable in regimes where Centralized training becomes unstable. We use higher learning rates to induce training instability in a controlled setting, following Wortsman et al. [1], who showed that raising the learning rate in smaller Transformers can reproduce specific instabilities observed at larger scales. At the lower rate in Figure 2, both protocols train stably and reach similar validation loss.

Q2 (Interpreting Figure 3) In Figure 3, for η = 1.5 and t = 1000–2000, the validation loss and Rayleigh quotient show different trends. This raises questions about how reliably the blockwise Rayleigh diagnostic represents the largest eigenvalue of the full Hessian.

A2. We respectfully note that sharpness matters for perturbation growth. In quadratic gradient descent, a perturbation along an eigendirection of curvature λ grows when ηλ > 2. Updates in other directions can still reduce loss, so the two trends can differ even with exact curvature. At η = 1.5, Centralized and Local SGD show unstable validation loss and larger blockwise Rayleigh values than gossip (Figure 3). Figure 3 measures curvature within principal blocks. The full-Hessian eigenvalue in Theorem 1 also accounts for cross-block couplings.

Q3 (Theory and Transformer training) The theory relies on deep linear networks, scalar inputs, squared loss, and small-noise conditions. … making the connection between theory and experiments unclear.

A3. Deep linear networks are a classical model for learning dynamics [2], nonconvex geometry [3], and instability beyond the edge of stability [4]. They let us isolate how worker deviations affect the mean update. Our theory covers more general settings in Appendices D, F, and G.

  • Models and data. D.4 and D.5 cover smooth losses with multiple curvature directions under local assumptions. F.2 and F.11 allow vector predictions and sample-dependent Hessians.
  • Communication and response. F.8 allows Markov mixing under conditional mean-square contraction of graph products. G.1 covers smooth nonlinear scalar predictors under local geometry and history conditions.
  • Initialization and time. Theorem 1 allows a full-parameter neighborhood of a conditional-mean fit. F.12 treats a fixed nonfitted start in a bilinear model. F.6 uses one small-noise threshold from step 2 to a local horizon of order 1/σ² when the tangential sharpness gradient is nonzero.

Figure 8 and Appendix H.5 test finite perturbations in a 124M AdamW Transformer. At peak LR 0.08, gossip's cumulative squared response over 20 steps is 6.4–40.9% of the Centralized response across four checkpoints and two amplitudes. Figure 8 and replay protocol

Q4 (H = 1 and theoretical intuition) What is the difference between Local SGD with H = 1 and centralized training? … the theorem statements are difficult to follow, with limited discussion of their intuition and implications.

A4. At H = 1, Local SGD clips locally, steps, and averages weights globally. Centralized training (DDP) averages gradients before clipping and stepping. Both use whole-model norm clipping at 1.0. The order can change the mean update (§4.2, Appendix C.1.3). A canonical example of clipping before the local step is DiLoCo [5] (§3).

Section 5 shows how first-order terms cancel in Theorem 1, leaving a quadratic correction (Figure 4). Theorem 2 states that the scalar trajectories follow one path at different speeds (Figure 5). Exchange timing can change the direction of motion with multiple outputs (Figure 6). Theorem 3 gives a computable condition, and Figure 7 connects the injection timeline to the response.

Q5 (Notation) Both N and T are used to represent training horizons in different sections.

A5. Thanks. We have addressed this.

References

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttal

We thank the AC for coordinating the review and all three reviewers for their careful reading and for suggesting controls and measurements that sharpen the comparison between training protocols. Reviewer VP8P highlighted the value of studying decentralized training through the lens of training-loss stability. Reviewers WQz1 and pMB6 recognized the breadth of the evidence across Transformer families, models up to 1.5B parameters, and three optimizers (AdamW, SGDM, and Muon), and noted the value of the SGDM global-averaging control. Reviewer WQz1 highlighted the theory's link between the mixing spectrum and the drift toward lower sharpness. Reviewer pMB6 emphasized the full-Hessian analysis at the worker-averaged iterate and the distinction between sharpness reduction and directional perturbation stability.

  • Matched local SGD control at H = 5 (WQz1, pMB6). Following your suggestion, we added a matched AdamW run with global parameter averaging every five steps (Local SGD). At peak LR 0.05, both Local SGD and pairwise gossip maintained stable validation loss through step 5,000. This extends the empirical finding to periodic global averaging in the tested setting. Validation curves
  • Direct directional perturbation test (WQz1, pMB6). We added a 124M AdamW Transformer experiment (Figure 8 and Appendix H.5). At peak LR 0.08, gossip's cumulative squared parameter response over 20 updates was 6.4–40.9% of the Centralized response across four checkpoints and two perturbation amplitudes. Figure 8 and replay protocol
  • Scope and intuition of the theory (VP8P, WQz1, pMB6). Section 5 uses Figures 4–7 to explain the mean update, the effect of communication timing, and the perturbation response. For SGD, the appendices also cover smooth vector predictions (Theorem F.2), time-varying Markov mixing (Theorem F.8), and smooth nonlinear scalar predictors (Theorem G.1). The SGD schedule that averages globally every five steps satisfies the contraction condition in Theorem F.8. Theory and interpretation
  • Update order and curvature measurements (VP8P, WQz1, pMB6). We clarified that Centralized training averages gradients before clipping, while Local SGD and gossip clip locally before stepping and later average parameters. Optimizer states stay local in both parameter-averaging protocols. The clipping order explains why the H = 1 SGDM update rules differ even with matched minibatches. We also specify the curvature batch and power-iteration settings and explain why blockwise curvature and validation loss can follow different trends.
  • Why test large learning rates (VP8P). Following Wortsman et al. [1], we use large learning rates to induce Transformer training instabilities in a controlled setting. This lets us test whether gossip keeps training stable in regimes where Centralized training exhibits loss spikes.

References

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.