- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper investigates how gossip communication affects optimization stability in language-model pre-training. Across several model families and optimizers, gossip-averaged models exhibit smoother validation-loss trajectories at aggressive learning rates. A deep-linear-network analysis establishes conditions for lower expected full-Hessian sharpness at the worker-averaged iterate, a mixing-dependent drift toward lower sharpness, and reduced finite-window sensitivity along a specified perturbation direction.
Critical strengths
The empirical observation is interesting and extends across AdamW, SGDM, Muon, several Transformer families, and models up to 1.5B parameters. The SGDM Local SGD comparison and three-seed reporting provide useful checks. A reproducible stability advantage in this setting would be valuable beyond the usual communication-efficiency motivation for gossip.
The theory provides a substantive, carefully delimited contribution. It analyzes the actual worker-averaged iterate and full Hessian, rather than only a surrogate curvature quantity, and separates sharpness reduction from directional perturbation stability. The explicit counterexamples and limitations make the sufficient mechanism easier to assess and prevent an unwarranted universal interpretation.
Critical weaknesses
The empirical attribution to gossip needs a more tightly matched control. Appendix C.1.3 selects H=5 for AdamW from four local-update horizons, while the reported Local SGD control uses SGDM with H=1. That control does not determine whether the main AdamW advantage requires pairwise gossip or also arises from global model averaging every five local updates. In addition, Appendix C.1.2 specifies clipping at norm 1.0 without making its order relative to gradient reduction explicit, and optimizer-state handling after model averaging is unclear. For paired minibatches, common initial momentum states, and no difference in nonlinear clipping, H=1 global parameter averaging under linear SGDM should reproduce the centralized mean update. The paper should clarify whether these conditions hold and whether the Figure 3 separation reflects protocol differences or sampling variability. These details matter directly to what phenomenon has been isolated.
The theory-to-Transformer connection remains suggestive. The theory uses a single prediction direction, fixed mixing, local small-noise conditions, and plain SGD, whereas the language-model runs use random pairings and momentum or adaptive optimization. The manuscript acknowledges these limits. However, the reported raw blockwise curvature does not show that the proved directional mechanism drives the observed stability. Appendix C.1.4 also leaves the curvature sample size, sharing rule, and power-iteration convergence settings unspecified, limiting the diagnostic’s reproducibility. This limits the explanatory claim and prevents a stronger recommendation.
Questions for authors
Could the authors report AdamW Local SGD at H=5 alongside gossip at H=5, and specify the exact order of reduction, clipping, optimizer-state updates, and parameter/state synchronization? Were minibatches paired and initial momentum states matched, and does the Figure 3 separation reflect different update operations or sampling variability?
What sample sets and power-iteration settings were used for the curvature measurements, and does the qualitative ordering persist with a fixed shared batch and a sufficiently converged estimate?
Response to Reviewer pMB6
Thank you for your support and for highlighting the full-Hessian analysis at the worker-averaged iterate. We appreciate your suggestion to include a matched Local SGD control.
A1. Thanks for suggesting this control. We ran a Local SGD-style AdamW control at H = 5, with global averaging every five local steps. We matched the gossip recipe in Figure 2(b) (124M model, peak LR 0.05, batch size, warmup, and local optimizer-state handling). The averaged models had stable validation curves at 25-step checkpoints through step 5,000, reaching 4.18 (Local SGD) and 4.24 (gossip). Each curve shows one run. The new control used a newer runtime. Matched validation curves
This result is consistent with our theory. Theorem F.8 (Appendix F.6) covers the corresponding SGD schedule. Worker deviations affect the mean update between rounds and can accelerate motion toward lower sharpness in the scalar small-noise model. This broadens the empirical scope to Local SGD at the tested setting.
A2. As in standard data parallelism, Centralized training (DDP) averages accumulated gradients before whole-model norm clipping at 1.0 and the optimizer step. Local SGD and Decentralized training (gossip) clip accumulated gradients locally at 1.0 and then step. At communication rounds, they average parameters globally or within pairs, respectively. Momentum buffers and AdamW moments stay local. Section 4.2 and Appendix C.1.3 give the update order.
Clipping before the local step is established practice. Representative examples are DiLoCo [1] (§3) for local updates and DECOR [2] (Algorithm 1) for gossip.
The Figure 3 runs use the same deterministic rank-wise minibatch schedule and start with zero momentum history. Each protocol has three repeats at nominal seed 41 with SGDM momentum 0.9. At H = 1, clipping the averaged gradient and averaging locally clipped gradients define different update rules even with paired batches. Local SGD and gossip share the local update rule and differ in how they average parameters. At peak LR 1.5, their validation curves separate in all three repeats, with mean losses of 34.00 for Local SGD and 3.96 for gossip at step 5,000.
A3. Each checkpoint uses 128 training sequences of length 1,024 (16 per worker). The batch stays fixed across all blocks and iterations within a measurement and advances between checkpoints. We use up to five warm-started iterations per block, following the exact-Hessian comparison in Ramasinghe et al. [3] (§4.2, Figure 2). With this protocol, Figure 3(c) shows smaller maximum absolute blockwise Rayleigh values for gossip at peak LR 1.5.
A4. Following your suggestion, we have added a directional perturbation experiment in a 124M Transformer trained with AdamW (Figure 8 and Appendix H.5). We inject perturbations along a fixed AdamW update direction and replay the next 20 updates. At peak LR 0.08, gossip has a smaller squared response at every replay step across four checkpoints and two finite amplitudes. Its 20-step cumulative response is 6.4–40.9% of the Centralized response. Both amplify the perturbation by step 20. Figure 8 and replay protocol
Our appendix theory covers more general SGD settings.
References