- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes SCAPE, a communication-efficient distributed optimizer for LLM pretraining that enables extreme sparse communication. SCAPE uses the first moment of AdamS to construct top- masks, distributes mask generation across optimizer shards to avoid the fill-in effect, and uses one-step delayed masks to overlap mask synchronization with computation. Experiments show that SCAPE can substantially reduce communication and training time while maintaining comparable model performance under high sparsity.
Critical strengths
-
The experimental results are strong. SCAPE achieves substantial communication and training-time reductions under very high sparsity while largely preserving the training behavior and model quality of dense baselines.
-
The empirical observations provide clear and convincing motivation for the algorithm design. In particular, the robustness of AdamS to aggressive sparsification and the temporal stability of first-moment-derived masks naturally motivate the choice of optimizer and the delayed-mask strategy.
-
The proposed design is well tailored to distributed LLM training. Aligning mask generation with optimizer sharding avoids the fill-in effect of conventional distributed top- sparsification, while delayed mask synchronization enables better overlap between communication and computation.
Critical weaknesses
-
I think that the theoretical analysis is somewhat limited. The convergence result is established only for an idealized fresh-mask variant rather than the practical delayed-mask SCAPE used in the experiments. It also relies on several nontrivial assumptions. Hence, the theory provides only limited justification for the convergence behavior of the actual algorithm.
-
The paper does not clearly isolate the contribution of error feedback, and the fresh-versus-delayed mask comparison is conducted only in a relatively limited setting. Additional ablations under more aggressive sparsity would better establish the robustness of these components.
Questions for authors
See weaknesses above.
2 comments
Thank you so much for your detailed responses to my concerns. I appreciate the clarifications and am happy to maintain my positive evaluation of this paper. Good luck!
We appreciate the reviewer's feedback, and we answer each of the reviewer's questions individually.
1. Limited theoretical analysis
We believe the theory gives more support than this summary suggests, for three reasons.
First, the assumptions are the standard ones for this class of methods. Smoothness, unbiased gradients with bounded variance, and bounded gradients are commonly used in the analysis of Adam-type optimizers. The contraction condition is the property under which error feedback is analyzed in the first place [EF]: top-k is a contraction operator, and Appendix F shows that it holds by construction for every selector's own proposal, with factor 1−d (proof of Lemma F.5). Assumption F.3 is its counterpart for a mask shared by all workers. In SCAPE the proposals of different workers share the first moment exactly (Algorithm 1, line 6), so the selecting worker's proposal differs from the average only through its local gradient, weighted by 1−β1=0.1, and its local residual. With a fixed preconditioner, the alignment and drift conditions are not needed at all (Proposition F.10).
Second, analyzing an idealized recursion is the norm and not a gap specific to SCAPE: convergence analyses of adaptive optimizers commonly omit gradient clipping, decoupled weight decay and learning-rate schedules, all of which are used in practice. Theorem 5.1 gives the standard O(T−1/2) stationarity rate for sparse communication with error feedback under AdamS, which is the mechanism SCAPE relies on.
Third, moving from fresh to delayed masks leaves the algebraic core of the proof intact: the transmitted and residual parts stay orthogonal, and the error-feedback identities hold unchanged (Remark F.12). The delayed mask is then supported empirically at every scale we train: the masks of adjacent steps overlap strongly (Section 3.2, Figure 3), at d = 0.1 fresh and delayed masks reach similar final losses (Figure 8b and the table under point 3), and every Llama-500M and Llama-1.8B run in Table 1 uses the delayed mask and trains without divergence; at d = 0.1 these runs end within 0.013–0.026 of dense AdamS in validation loss.
2. Ablation on error feedback
In the background section (Section 2.2), we have cited the prior work that ablates the effectiveness of using error feedback (EF) for DL model training with compressed communication. Since SCAPE is designed based on AdamS, which is different from prior works, we ablate the importance of using EF in Sec 6.3. From Figure 8 and Figure 15, we can see that without EF, SCAPE has much higher train and validation losses compared to dense AdamS (0.77 higher in validation loss at d = 0.1 on GPT-345M). EF is therefore already indispensable at d = 0.1, and at d = 0.01 only a tenth as many entries are transmitted per step, so we did not repeat this ablation at higher sparsity.
3. Ablation on d = 0.01 for fresh-vs-stale mask
We agree that using higher sparsity to ablate the fresh-versus-delayed mask can strengthen the ablation study, and we have run this ablation on GPT-345M at both densities. The final training and validation losses of the four runs are listed in the table below. At d = 0.1, the fresh and the delayed mask end within 0.008 of each other in both training and validation loss, so the one-step delay has no material effect on quality at the setting on which our conclusions rest. At d = 0.01, the fresh mask ends 0.055 lower in validation loss (3.0065 against 3.0614), so the delay has a cost at this extreme sparsity. This is one of the reasons why we now use d = 0.1 as the default setting and present d = 0.01 as a limit test. We have revised the abstract, introduction and conclusion to state the speedups for d = 0.1. The delayed mask remains part of the system design rather than an approximation made for convenience: a fresh mask must be built from the current step's scores and all-gathered before that step's sparse exchange, so selection and the mask all-gather must finish before any worker can pack its values, whereas the one-step delay lets the sparse exchange start from a mask that every worker already holds. The delay is also limited by construction. An entry that the previous step's mask leaves out stays in the residual, and the next score is this carried state plus one new gradient weighted by 1−β1=0.1 (Algorithm 1, lines 6–9), so a stale mask postpones an entry by one step at any density and does not drop it.
[EF] Sebastian U. Stich, Jean-Baptiste Cordonnier, and Martin Jaggi. Sparsified SGD with memory. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS’18, pp. 4452–4463, Red Hook, NY, USA, 2018. Curran Associates Inc.