acceptodds
Under review as a conference paper at ICLR 2027

SCAPE: Accurate and Efficient LLM Training with Extreme Sparse Communication

Abstract

Communication increasingly dominates the cost of Large Language Model (LLM) pre-training, especially under data-parallel and sharded training schemes, where gradient synchronization and parameter reconstruction overhead increase with model size and system scale. Existing communication-reduction methods either sparsify raw gradients, which can be unstable for modern Adam-style optimizers at high sparsity, or quantize communication, whose savings are fundamentally bounded by bit width. We present SCAPE, a communication-efficient distributed optimizer for LLM training that exploits the stability of AdamS's first-moment to enable aggressive sparsification while preserving LLM quality. Instead of constructing masks from gradients, SCAPE derives them from first-moment-based statistics, partitions mask generation across workers to align with optimizer sharding, and delays mask usage by one step so that mask synchronization can overlap with computation. SCAPE also reconstructs the quantities required for second-moment updates from a single synchronized sparse buffer, avoiding an additional collective. We implement SCAPE in Megatron-LM and evaluate its convergence by pre-training GPT-345M on OpenWebText and Llama-500M on SlimPajama-6B using 32 NVIDIA GH200 GPUs on TACC Vista. In both models, SCAPE preserves training stability, validation loss, and downstream task accuracy under 90% and 99% sparsity. For Llama-500M, SCAPE reduces end-to-end pre-training wall-clock time by up to 43.3% while maintaining model quality comparable to dense AdamW and AdamS. For Llama-1.8B, SCAPE achieves up to 3.20 speedup per step compared to dense distributed AdamS.

Primary area:
optimization
Keywords:
Distributed Computing, Communciation Compression, Sparsity

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_U8YQOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SCAPE, a communication-efficient distributed optimizer for LLM pretraining that enables extreme sparse communication. SCAPE uses the first moment of AdamS to construct top-kk masks, distributes mask generation across optimizer shards to avoid the fill-in effect, and uses one-step delayed masks to overlap mask synchronization with computation. Experiments show that SCAPE can substantially reduce communication and training time while maintaining comparable model performance under high sparsity.

Critical strengths
  • The experimental results are strong. SCAPE achieves substantial communication and training-time reductions under very high sparsity while largely preserving the training behavior and model quality of dense baselines.

  • The empirical observations provide clear and convincing motivation for the algorithm design. In particular, the robustness of AdamS to aggressive sparsification and the temporal stability of first-moment-derived masks naturally motivate the choice of optimizer and the delayed-mask strategy.

  • The proposed design is well tailored to distributed LLM training. Aligning mask generation with optimizer sharding avoids the fill-in effect of conventional distributed top-kk sparsification, while delayed mask synchronization enables better overlap between communication and computation.

Critical weaknesses
  • I think that the theoretical analysis is somewhat limited. The convergence result is established only for an idealized fresh-mask variant rather than the practical delayed-mask SCAPE used in the experiments. It also relies on several nontrivial assumptions. Hence, the theory provides only limited justification for the convergence behavior of the actual algorithm.

  • The paper does not clearly isolate the contribution of error feedback, and the fresh-versus-delayed mask comparison is conducted only in a relatively limited setting. Additional ablations under more aggressive sparsity would better establish the robustness of these components.

Questions for authors

See weaknesses above.

2 comments

We appreciate the reviewer's feedback, and we answer each of the reviewer's questions individually.

1. Limited theoretical analysis

We believe the theory gives more support than this summary suggests, for three reasons.

First, the assumptions are the standard ones for this class of methods. Smoothness, unbiased gradients with bounded variance, and bounded gradients are commonly used in the analysis of Adam-type optimizers. The contraction condition is the property under which error feedback is analyzed in the first place [EF]: top-k is a contraction operator, and Appendix F shows that it holds by construction for every selector's own proposal, with factor 1−d\sqrt{1-d} (proof of Lemma F.5). Assumption F.3 is its counterpart for a mask shared by all workers. In SCAPE the proposals of different workers share the first moment exactly (Algorithm 1, line 6), so the selecting worker's proposal differs from the average only through its local gradient, weighted by 1−β1=0.11-\beta_1 = 0.1, and its local residual. With a fixed preconditioner, the alignment and drift conditions are not needed at all (Proposition F.10).

Second, analyzing an idealized recursion is the norm and not a gap specific to SCAPE: convergence analyses of adaptive optimizers commonly omit gradient clipping, decoupled weight decay and learning-rate schedules, all of which are used in practice. Theorem 5.1 gives the standard O(T−1/2)O(T^{-1/2}) stationarity rate for sparse communication with error feedback under AdamS, which is the mechanism SCAPE relies on.

Third, moving from fresh to delayed masks leaves the algebraic core of the proof intact: the transmitted and residual parts stay orthogonal, and the error-feedback identities hold unchanged (Remark F.12). The delayed mask is then supported empirically at every scale we train: the masks of adjacent steps overlap strongly (Section 3.2, Figure 3), at dd = 0.1 fresh and delayed masks reach similar final losses (Figure 8b and the table under point 3), and every Llama-500M and Llama-1.8B run in Table 1 uses the delayed mask and trains without divergence; at dd = 0.1 these runs end within 0.013–0.026 of dense AdamS in validation loss.

2. Ablation on error feedback

In the background section (Section 2.2), we have cited the prior work that ablates the effectiveness of using error feedback (EF) for DL model training with compressed communication. Since SCAPE is designed based on AdamS, which is different from prior works, we ablate the importance of using EF in Sec 6.3. From Figure 8 and Figure 15, we can see that without EF, SCAPE has much higher train and validation losses compared to dense AdamS (0.77 higher in validation loss at dd = 0.1 on GPT-345M). EF is therefore already indispensable at dd = 0.1, and at dd = 0.01 only a tenth as many entries are transmitted per step, so we did not repeat this ablation at higher sparsity.

3. Ablation on dd = 0.01 for fresh-vs-stale mask

We agree that using higher sparsity to ablate the fresh-versus-delayed mask can strengthen the ablation study, and we have run this ablation on GPT-345M at both densities. The final training and validation losses of the four runs are listed in the table below. At dd = 0.1, the fresh and the delayed mask end within 0.008 of each other in both training and validation loss, so the one-step delay has no material effect on quality at the setting on which our conclusions rest. At dd = 0.01, the fresh mask ends 0.055 lower in validation loss (3.0065 against 3.0614), so the delay has a cost at this extreme sparsity. This is one of the reasons why we now use dd = 0.1 as the default setting and present dd = 0.01 as a limit test. We have revised the abstract, introduction and conclusion to state the speedups for dd = 0.1. The delayed mask remains part of the system design rather than an approximation made for convenience: a fresh mask must be built from the current step's scores and all-gathered before that step's sparse exchange, so selection and the mask all-gather must finish before any worker can pack its values, whereas the one-step delay lets the sparse exchange start from a mask that every worker already holds. The delay is also limited by construction. An entry that the previous step's mask leaves out stays in the residual, and the next score is this carried state plus one new gradient weighted by 1−β1=0.11 − \beta_1 = 0.1 (Algorithm 1, lines 6–9), so a stale mask postpones an entry by one step at any density and does not drop it.

DensityMaskTrain lossVal. loss
d = 0.1Delayed3.0112.955
d = 0.1Fresh3.0042.947
d = 0.01Delayed3.1223.061
d = 0.01Fresh3.0663.007

[EF] Sebastian U. Stich, Jean-Baptiste Cordonnier, and Martin Jaggi. Sparsified SGD with memory. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS’18, pp. 4452–4463, Red Hook, NY, USA, 2018. Curran Associates Inc.

Thank you so much for your detailed responses to my concerns. I appreciate the clarifications and am happy to maintain my positive evaluation of this paper. Good luck!

Reviewer_EDPSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SCAPE is a data-parallel optimizer that sparsifies the communicated AdamS state instead of raw gradients. Each worker picks the top-k coordinates of a first-moment score on its own shard, the shard masks are all-gathered into one shared mask, and the mask is applied one step later so that its exchange can overlap with computation. Only masked values are all-reduced; the rest stays in a local error-feedback buffer, and the second-moment input is rebuilt from the same buffer. A distributed-optimizer (DO) variant uses two sparse all-reduces. Experiments pretrain GPT-345M and Llama-500M on 32 GH200 GPUs at 90% and 99% sparsity, measure step times for Llama-1.8B on up to 64 GPUs, and compare final loss with five compression methods; ablations and a convergence result under stated assumptions complete the paper.

Critical strengths
  1. A practical design that removes known obstacles to top-k communication, measured end to end. Shard-owned masks keep the density at its target by construction (0.1 at 32 GPUs, while DGC and LEGACY fill in to roughly 0.9 and 0.55, Fig. 7); the delayed mask, which lets the mask exchange overlap with computation, reaches a final loss similar to a fresh mask (Fig. 8b); and the Megatron-LM implementation cuts Llama-500M training time from 2.47 to 1.40 days (Table 1). Wall-clock results, not only byte counts, make this directly useful.

  2. A useful observation about the optimizer: under shared hyperparameters, AdamS tolerates sparsified error feedback much better than AdamW (Figs. 1 and 8a), and with AdamS the validation loss stays within 0.03 of the dense runs at 90% and 99% sparsity (Table 1). If this holds with per-optimizer tuning, the choice of optimizer is a simple lever for aggressive compression.

Critical weaknesses
  1. The efficiency gains over existing compression methods are not established. The speedups over the tested dense baselines are real, but the largest one (3.20x per step for Llama-1.8B) comes from a communication-dominated 64-GPU run, and the paper does not report achieved collective throughput, how the compute/communication split in Fig. 15 was measured, or the dense baseline's overlap settings. Sec. 6.2 explains why the compression baselines were not timed (plain PyTorch implementations against SCAPE's CUDA kernels). That is fair, but it leaves runtime gains over methods such as PowerSGD and SDP4Bit, which reach similar plotted loss (Fig. 7), unestablished. There is also a memory cost: in a separate 4-GPU, d = 0.1 measurement, SCAPE + DO uses 75,832 MB per GPU versus 47,980 MB for AdamS + DO (69,474 MB with residual offload; Fig. 16b). Matched measurements of bytes, step time and peak memory against a representative optimized compressor would show the practical advantage.

  2. The quality evidence is narrow in scale and uncertainty. Quality is evaluated at 345M and 500M; the 1.8B runs are timing measurements only. App. F implies about 419B training tokens for Llama-500M on SlimPajama-6B, a corpus of roughly 6B tokens, so about 70 passes if the batch size counts sequences. The shorter 7.86B-token GPT-345M comparison gives useful evidence with far less reuse, but only at 90% sparsity; the 99% results come from runs with repeated data (about 5.8 passes for GPT-345M and 70 for Llama-500M). Run counts and uncertainty are not reported, and some downstream scores drop materially (Llama-500M BoolQ 51.90 at d = 0.1 vs 57.65 for AdamW). The convergence result is explicitly conditional and does not cover the delayed-mask implementation that is run (Sec. 5, App. E). Together these limit the support for the broad claim that quality is preserved at 99% sparsity.

Questions for authors
  1. How was the compute/communication split in Fig. 15 measured, and what collective dtypes, bucket sizes, overlap settings (gradient reduce, parameter gather) and achieved collective throughput apply to the dense and SCAPE runs from 4 to 64 GPUs? By a rough estimate (ideal ring, FP32 reduce-scatter, BF16 all-gather), 3.1 s of communication in the 3.9 s dense Llama-1.8B step implies about 3.4 GB/s per GPU, far below the nominal 400 Gbps, so evidence that the dense baseline is tuned would make the speedups more convincing. Could you also report peak memory for the 64-GPU runs behind the headline speedups?

  2. Could you compare against a representative optimized compressor, for example SDP4Bit, which is not among the plain-PyTorch implementations named in Sec. 6.2, reporting step time, compression overhead and achieved quality? Please also report steady-state and cumulative bytes, including warmup, uncompressed tensors, the mask, padding and dtypes, so that "equivalent compression rate" (Sec. 6.2) is quantified; by my rough factor count, PowerSGD at rank 16 sends about 2% of the dense values on GPT-345M in steady state.

  3. Please confirm the token budgets, dataset versions and passes implied by App. F, and the number of independent runs behind each result. Is there quality evidence at 99% sparsity with less data reuse, or at the 1.8B scale, for example from existing checkpoints or a focused matched-budget run?

5 comments

We appreciate the reviewer's constructive comments and provide answers to them below. We have also updated our manuscript with revisions. Please refer to the latest version for the updates mentioned below.

1. Measurement, settings, and achieved throughput

The reviewer's calculation is correct: the profiling runs in the submission used NCCL's TCP socket transport for inter-node communication, at about 27 Gbps per GPU.

Due to the limited computation resources on Vista, we performed the profiling again on TACC Horizon using 16 to 128 GB200 GPUs, at the global batch of our training runs (global batch size of 1,024 and sequence length of 4K tokens). We use the socket transport to model a bandwidth-limited environment, such as the 10 Gbps and 5 Gbps networks used in DUO [DUO], while the intra-node network (four GPUs per node) still uses NVLink. The detailed methods used for profiling are summarized in Appendix A. Specifically, we use FP32 for the gradient reduce-scatter and BF16 for the parameter all-gather. The time for computation, gradient synchronization, and parameter update is profiled using Megatron-LM's level-2 timers, and the reported time is averaged over steps 21–40, where steps 1–20 are used for warming up the machine. When overlapping is off (Figure 6 and the overlap-off bars of Figure 9), all gradients form one bucket; this is also the setting of every SCAPE run, which has no overlap path. With overlapping on for both the gradient reduce-scatter and the parameter all-gather (Figure 9), we use the default bucket size in Megatron-LM: max(40M, 1M x data-parallel size). Concretely, the bucket size is 40M at 16 and 32 GPUs, 64M at 64, and 128M at 128. Enabling overlapping shortens the dense step by only 2–12% (Figure 9). Even against the overlapped dense baseline, SCAPE's speedup for Llama-8B on 128 GPUs remains 3.68x (dd = 0.1). We also provide the achieved throughput in Table 3 in Appendix A: the dense collectives run at 37–47 Gbps per node at every model size and GPU count, which is the capacity of this transport, so the dense baseline is limited by the network and is not under-tuned. The speedup follows the communication share, as expected for a method that reduces communication, and Figure 6 reports the whole range rather than a single point: communication takes 36–42% of the dense step at 16 GPUs and 83–84% at 128 GPUs (Figure 9) and SCAPE (dd = 0.1) is correspondingly 1.4–1.6x faster at 16 GPUs and 3.9–4.2x faster at 128 GPUs. SCAPE is faster than dense AdamS at every measured point. These measurements are at about 40 Gbps per node; on a 10 Gbps network, where the dense exchange takes four times as long, they imply a speedup of at least 3.5x for SCAPE (dd = 0.1) for Llama-1.8B on 32 GPUs, instead of 1.88x.

The peak allocated memory for every profiled experiment (sequence length 4K, global batch size 1K) is plotted in Figure 6g, 6h, 6i, including training Llama-1.8B, Llama-7B, and Llama-8B on 16, 32, 64, and 128 GPUs. On the memory cost: for Llama-8B on 128 GPUs, dense AdamS peaks at 126.9 GB, SCAPE at 143.0 GB (dd = 0.1) and 131.9 GB (dd = 0.01), and PowerSGD at 157–158 GB. SCAPE (dd = 0.1) therefore adds 13% to the memory of dense AdamS, about half of what PowerSGD adds (24%). This overhead shrinks as GPUs are added (from 23.0 GB at 16 GPUs to 16.1 GB at 128 for dd = 0.1), while PowerSGD's stays at 31 GB. The larger gap in the previous manuscript comes from an implementation that kept a full FP32 parameter replica and a full FP32 residual buffer, which the profiled implementation no longer does.

2. Llama-500M and Llama-1.8B training results

Reviewer's calculation about the number of data reuse is correct: we notice the high data reuse for training Llama-500M on SlimPajama-6B, so we have pre-trained another set of Llama-500M models for 41.9B tokens from FineWeb-Edu [FineWeb-Edu] (sample-100BT, 0.385 of one pass). We have also pre-trained Llama-1.8B from scratch using 73.4B tokens from the same corpus (sample-350BT, 0.193 of one pass). The hyperparameters used to train these two models can be found in Appendix G. Both models are trained without data reuse, and their loss curves can be found in Figure 5 (validation losses) and Figure 14 (training losses). Every configuration is trained once, and the uncertainties in the tables are standard errors over the questions. From the final training and validation losses for GPT-345M, Llama-500M, and Llama-1.8B trained using dense AdamS, SCAPE (dd = 0.1), and SCAPE (dd = 0.01) presented in Table 1, we can see that SCAPE (dd = 0.1) preserves the model quality for all models (its final validation loss is 0.026 higher for Llama-500M and 0.013 higher for Llama-1.8B), while SCAPE (dd = 0.01) has higher losses (0.044 and 0.066 higher). We also evaluate the quality of models trained using different approaches on extensive downstream tasks (Table 2), and we notice that without data reuse Llama-500M does not have the same degradation issue on tasks like BoolQ and COPA. Instead, Llama-500M trained using SCAPE (dd = 0.1) scores above dense AdamS on BoolQ, CB and COPA and within 1.2 points of it on the other five tasks. We notice that for Llama-1.8B, SCAPE (dd = 0.01) causes performance degradation on specific tasks, such as LAMBADA (39.24 against 42.75), ARC-E (64.10 against 66.62) and HellaSwag (38.85 against 40.74). We therefore now use dd = 0.1 as the default setting and present 99% sparsity as a limit test: on single-pass data at two scales, SCAPE (dd = 0.01) trains stably but ends 0.044 and 0.066 above dense AdamS in validation loss and loses accuracy on several downstream tasks. Our claims on preserved quality and on speedup rest on dd = 0.1, and we have revised the abstract, introduction and conclusion to state them for this setting. The convergence result is stated as conditional by design, like the error-feedback analyses it builds on; its algebra carries over to the delayed mask (Remark F.12), and every Llama-500M and Llama-1.8B SCAPE run in Table 1 uses the delayed mask and trains without divergence.

3. Comparison to SDP4Bit and PowerSGD

We include the step-time comparison with SDP4Bit and PowerSGD (our own implementation, with effective compression rates of 10x and 100x for the gradient) in Figure 6. All methods are tested in the same network environment and on the same nodes. The rank in PowerSGD used for each layer in Llama-1.8B, Llama-7B, and Llama-8B can be found in Appendix G.4. SDP4Bit's step time is its measured exchange added to our dense computation, because the authors' Megatron-LM code base computes more slowly than ours. The timed exchange of every method includes its compression kernels, so the compression overhead is inside the reported times. We would like to emphasize that SDP4Bit quantizes each gradient value exchanged across nodes to 4.5 bits, a compression rate of around 7.1x for that exchange, which is lower than the 10x of SCAPE's gradient values at 90% sparsity. SDP4Bit-G1, which quantizes these gradient values to 1 bit, is a baseline in the DUO [DUO] paper. We could not find a public implementation of it, so we do not include it in the step-time, scaling efficiency, and memory tests. For Llama-8B on 128 GPUs at a global batch of 1,024, the speedup over dense AdamS is 4.09x for SCAPE (dd = 0.1), 3.73x for SDP4Bit and 1.74x for PowerSGD (dd = 0.1). Across Figure 6, SCAPE (dd = 0.1) is level with SDP4Bit up to 32 GPUs and about 5–17% faster at 64 and 128 GPUs, while sending half of its bytes. In exchange time alone, SCAPE (dd = 0.1) is 7–8x shorter than dense AdamS for Llama-8B, against 6.0–6.7x for SDP4Bit and at most 2.9x for PowerSGD, so its lead over both grows with the GPU count (over SDP4Bit, from 1.00x at 16 GPUs to 1.10x at 128).

Relative to dense AdamS, SCAPE sends 12.4% (dd = 0.1) and 3.4% (dd = 0.01) of the bytes per step, including the 1-bit mask, and 20.7% and 11.3% over the whole Llama-1.8B run, including the 1,000 dense steps and the density warmup. Padding, the selection-threshold histogram and the uncompressed RMSNorm parameters add under 0.1 percentage points. PowerSGD sends 34–40%, because it still all-gathers the full BF16 weights, and SDP4Bit and SDP4Bit-G1 about 24.2% and 22.6% (with their 4-bit weight update and 8-bit intra-node communication counted). The reviewer's count for Section 6.2 is right: that PowerSGD run used rank 16 and sent about 2.1% of the dense gradient values, which does not align with the ranks used in training GPT-345M. We have corrected it in Section 6.2 and provided the real ranks used for each layer in GPT-345M in Appendix G.4 (Table 4). The per-step communication volume comparison is summarized in the table below (bits that each GPU sends per parameter per step on 32 GPUs, i.e. 8 nodes x 4 GPUs, where "refresh" is a dense BF16 parameter refresh every 100 steps):

MethodIntra-nodeGradient across GPUsWeightsMask + refreshTotal% of dense
Dense AdamS–31.015.5–46.5100%
PowerSGD (d = 0.1)–3.115.5–18.640.0%
PowerSGD (d = 0.01)–0.3115.5–15.834.0%
SDP4Bit6.40.983.9–11.224.2%
SDP4Bit-G16.40.273.9–10.522.6%
SCAPE (d = 0.1)–3.11.551.125.812.4%
SCAPE (d = 0.01)–0.310.151.121.63.4%

Because no public code is available for SDP4Bit-G1, we extended the authors' SDP4Bit code to 1-bit gradients across nodes (1.25 bits per value with its scales), left the rest of SDP4Bit unchanged, and trained it with AdamS on our recipes. The SDP4Bit-G1 reported here is therefore our own implementation. We provide its quality comparison in the table below. Gaps are computed from the unrounded values, and the dense AdamS run of the SDP4Bit code base ends 0.006 above ours. With 22.6% of the dense bytes, SDP4Bit-G1 is behind SCAPE (dd = 0.1, 12.4%) at both scales: it ends 0.081 above dense AdamS against 0.026 for Llama-500M, and 0.061 against 0.013 for Llama-1.8B. Even at its limit setting, SCAPE (dd = 0.01) reaches a lower loss for Llama-500M (0.044 against 0.081) and a comparable one for Llama-1.8B (0.066 against 0.061) with 3.4% of the bytes, about one seventh of SDP4Bit-G1's. We believe the smaller difference at Llama-1.8B is caused by the lr scheduler: with the cosine lr scheduler of Llama-1.8B, SDP4Bit-G1 gradually catches up with SCAPE (dd = 0.01) as the lr decreases, while Llama-500M uses a constant lr.

Reviewer_u4w2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SCAPE reduces communication in distributed LLM training through AdamS-based sparsification, sharded common-mask selection, delayed mask synchronization, and sparse state reconstruction. Experiments report comparable validation losses at 90% and 99% sparsity on 345M/500M models, up to 43.3% lower end-to-end training time on 500M, and 3.20× per-step speedup on 1.8B.

Critical strengths

S1: The common-mask and sparse parameter-update design reduces both gradient and parameter communication under optimizer sharding, addressing a practical bottleneck in distributed training.

S2: Complete 345M/500M training runs provide useful empirical evidence. The reported 43.3% reduction in 500M training time with close validation loss demonstrates meaningful practical savings.

Critical weaknesses

W1: The incremental contribution is insufficiently established. Shared-mask selection and temporal mask reuse have precedents in ScaleCom and Radius, yet matched AdamS-based controls are absent. DUO (NeurIPS 2025) already combines compression with communication–computation overlap for sharded LLM training, but is neither discussed nor evaluated. Section 6.2 also omits runtime comparisons with compression baselines, leaving SCAPE’s advantage in time to matched quality unestablished.

W2: Quality preservation is not demonstrated at the setting supporting the largest speedup: 1.8B has no convergence or downstream results. Small-model evaluations lack repeated-run uncertainty despite substantial task-level drops, including BoolQ falling from 57.55 to 51.90 against dense AdamS at d=0.1 (Table 2).

Questions for authors

Q1: Under matched hardware, optimizer, token budget, and gradient-accumulation settings, how does SCAPE compare with DUO in time to matched quality, total communication volume, and peak memory? Which gains specifically require SCAPE’s state reconstruction and mask design?

Q2: At 1.8B and d=0.01, does SCAPE preserve dense-model quality under the same token budget? Please quantify uncertainty in the downstream evaluations and establish whether the drops in Table 2 are reproducible.

4 comments

We appreciate the reviewer's insightful comments. Here, we answer each of the reviewer's questions individually. We have also updated our manuscript with revisions. Please refer to the latest version for the updates mentioned below.

1. Novelty (compared to ScaleCom and Radius)

SCAPE differs from both methods in how the mask is built and in what is compressed. ScaleCom uses a single leading worker to build the mask at each step and rotates this role among the workers in cyclic order. The leading worker ranks its own error-compensated gradient over the whole model and broadcasts the top-k indices. No other worker can compress anything until these indices arrive. The selection is therefore centralized, serial work on the critical path: on NN GPUs, N−1N-1 of them sit idle while a single GPU ranks all the parameters, and the mask reflects a single worker's gradient. For Llama-8B on 128 GPUs, one GPU would rank all 8.0 billion coordinates at every step while 127 wait. In SCAPE, the NN shard owners select concurrently, each over only the 1/N1/N of the parameters that it owns (63 million per GPU in the same example), so the selection is NN-fold parallel: no GPU waits for another, and the cost per GPU falls as GPUs are added. Therefore, SCAPE's top-k selection is load-balanced and aligned with the distributed optimizer.

Radius avoids the cost of selection by re-selecting only every TT = 200–400 steps, each time after a dense all-reduce, so its mask is up to hundreds of steps old, and it applies the sparse gradient to AdamW, for which, as its paper states, the accumulated error cannot recover the optimizer state. It reports that quality is preserved at dd = 0.4, and that at dd = 0.1 the final validation perplexity is much higher than the baseline's and the training loss is unstable. SCAPE removes both limits: its distributed selection makes a refresh at every step affordable, so the mask is never more than one step old, and its state reconstruction on AdamS recovers the averaged update from the synchronized buffer. SCAPE therefore stays within 0.026 of dense AdamS in validation loss at dd = 0.1 (Table 1), the density at which Radius reports a much higher perplexity and an unstable training loss, and four times below the density at which Radius preserves quality. As a limit test, SCAPE still trains stably at dd = 0.01.

On matched controls: ScaleCom and Radius compress only the gradient, so an AdamS-based version of either would still exchange the full weights under the distributed optimizer. Figure 6 measures this limit with a gradient-only compressor in our own code base: PowerSGD keeps the dense weight all-gather (3.2 s of the 11.2 s dense step for Llama-8B on 128 GPUs) and saturates at 2.31x even at 100x gradient compression, whereas SCAPE reaches 4.09x at only 10x (dd = 0.1). In bytes, a gradient-only compressor cannot go below one third of the dense volume (the BF16 weights), whereas SCAPE (dd = 0.1) sends 12.4%. The comparison that decides the outcome is therefore gradient-only compression against sparsifying the optimizer state, which Figure 6 provides; for this reason we did not re-implement the two selection rules on AdamS.

SCAPE’s novelty lies in two designs. First, the state reconstruction: the averaged gradient and the residual are recovered from the single synchronized sparse buffer (Algorithm 1, lines 8–12), so the second-moment update needs no further exchange. Second, the mask acts on the optimizer state, so only masked parameters change, apart from a deterministic weight decay (Eq. 3).

Moreover, as the profiling results in Figure 9 show, the weight all-gather accounts for 12–28% of the dense step. SCAPE sparsifies it, whereas neither ScaleCom nor Radius can accelerate it. We have cited ScaleCom in the related work section (Appendix C).

2. Comparison to DUO

SCAPE and DUO are complementary approaches. The most important difference between DUO and SCAPE is that DUO uses the computation of the next step to hide the communication time of synchronizing a stale FP32 gradient, while SCAPE delays only the mask and sends the fresh gradient. In fact, DUO does not reduce the gradient traffic, because it sends the full-precision gradient on top of the provisional gradient quantized by SDP4Bit. By our count from its Algorithm 1 (a full-precision gradient, a 4-, 1- or 0-bit gradient, and 4-bit weights), DUO's total communication volume is about 75–90% of dense training, against 12.4% for SCAPE (dd = 0.1). For time, the DUO paper states its condition: each iteration's computation time must exceed its high-precision gradient communication time. In our socket profiles (Figure 9) the FP32 gradient reduce-scatter takes 0.4–1.0x the computation at 16 and 32 GPUs and 1.7–3.4x at 64 and 128 GPUs. Where the condition holds, DUO reports a throughput comparable to SDP4Bit (the baseline used in the DUO paper). Figure 6 measures the official implementation of SDP4Bit from its authors on our nodes, and at 16 and 32 GPUs, where the condition holds, SCAPE (dd = 0.1) is level with SDP4Bit, with half of its bytes. Where the condition fails, DUO's own Table 5 shows its throughput falling to 47–65% of SDP4Bit's when the gradient communication takes 2.1–2.8 times the computation, whereas SCAPE (dd = 0.1) is about 5–17% faster than SDP4Bit at 64 and 128 GPUs. In that regime DUO's iteration cannot be shorter than its full-precision reduce-scatter, which alone takes 6.2 s for Llama-8B on 128 GPUs, against 2.7 s for SCAPE's whole step (dd = 0.1): SCAPE compresses the communication itself instead of hiding it. For memory, DUO reports a footprint close to SDP4Bit's and keeps the full-precision gradient in host memory. SCAPE's peak memory at dd = 0.1 is 13% above dense AdamS for Llama-8B on 128 GPUs (Figure 6i) and needs no host-memory buffer. We could not find a public implementation of DUO, so this comparison through its own baseline is the closest matched measurement available. We have added DUO to the related work (Appendix C) and properly cited it, but the evaluation of combining DUO and SCAPE is beyond the scope of this work, so we leave it as future work.

3. Llama-1.8B Pretrain Experiment

We provide the losses and downstream scores of the pre-trained 1.8B model in Table 1, Table 2, Table 7, and Figure 5c. Every configuration is trained once, and the uncertainties in the tables are standard errors over the questions of each task. The hyperparameters for Llama-1.8B model training mostly follow H2O-Danube-1.8B [H2O], and the detailed hyperparameters can be found in Appendix G.3. All Llama-1.8B models are trained with 73.4B tokens from the FineWeb-Edu corpus [FineWeb-Edu] without data reuse. SCAPE (dd = 0.1) matches dense AdamS: its validation loss is within 0.013, its average over the eight tasks of Table 2 is 55.06 against 54.65, and seven of the eight tasks are within one standard error. We include SCAPE (dd = 0.01) as a limit test of how far the exchange can be sparsified, and use dd = 0.1 as the default setting: under the same token budget, the validation loss of SCAPE (dd = 0.01) is 0.066 higher, and LAMBADA (39.24 against 42.75), ARC-E (64.10 against 66.62) and HellaSwag (38.85 against 40.74) drop, while five of the eight tasks remain within one standard error. Our conclusions on quality and speed therefore rest on dd = 0.1, and we have revised the abstract, introduction and conclusion to state them for this setting. We also trained Llama-500M again on FineWeb-Edu for 41.9B tokens without data reuse, and the drop in downstream performance is not reproduced: BoolQ is 60.92 for SCAPE (dd = 0.1) against 59.39 for dense AdamS, and no task of Table 2 is more than one standard error below dense AdamS at dd = 0.1. The earlier runs repeated SlimPajama-6B about 70 times.

4. Runtime against compression baselines and time to matched quality.

Figure 6 reports per-step time, strong-scaling efficiency, and peak memory for PowerSGD (our own implementation) and SDP4Bit, measured on the same nodes and network from 16 to 128 GPUs. For Llama-8B on 128 GPUs at a global batch of 1,024 and sequence length of 4K, the speedup over dense AdamS is 4.09x for SCAPE (dd = 0.1), against 3.73x for SDP4Bit and 1.74x for PowerSGD at the same gradient compression (dd = 0.1); even at 100x gradient compression (dd = 0.01), PowerSGD reaches only 2.31x. SCAPE (dd = 0.1) shortens the exchange itself by 7–8x for Llama-8B, against 6.0–6.7x for SDP4Bit and at most 2.9x for PowerSGD. Because the computation shrinks with the GPU count while the exchange does not, each method's speedup tends to this ratio, so SCAPE's lead grows with scale: at 128 GPUs SCAPE (dd = 0.1) has reached 4.09x of a possible 7.4x, while PowerSGD (dd = 0.1) is at 1.74x of 2.0x. Table 1 provides the end-to-end time of Llama-500M and Llama-1.8B at an equal number of steps, computed from the per-step time measured on 32 GPUs with the same number of tokens per step: SCAPE (dd = 0.1) is 1.40x faster for Llama-500M and 1.63x faster for Llama-1.8B, which saves 225 GPU-hours per Llama-1.8B run. For matched quality, dense AdamS reaches the final validation loss of SCAPE (dd = 0.1) after about 85% of SCAPE's steps. With the measured step times, SCAPE (dd = 0.1) therefore reaches its final loss sooner than dense AdamS reaches the same loss at every GPU count: 11.1 h against 15.4 h on 32 GPUs (28% less time), and 43% and 55% less time on 64 and 128 GPUs. At the limit setting dd = 0.01, dense AdamS reaches SCAPE's final validation loss after about half of SCAPE's steps, so SCAPE (dd = 0.01) pays off in time only from 64 GPUs on: it needs 14% more time on 32 GPUs and 14% and 33% less on 64 and 128 GPUs. Our socket transport still delivers about 40 Gbps per node. On a 10 Gbps network, such as the low-bandwidth setting of DUO, the dense exchange takes four times as long, and the same measurements imply a per-step speedup of at least 3.5x for SCAPE (dd = 0.1) for Llama-1.8B on 32 GPUs. SCAPE (dd = 0.1) would then reach matched quality in about half the time of dense AdamS on 32 GPUs.

[H2O] Philipp Singer, Pascal Pfeiffer, Yauhen Babakhin, Maximilian Jeblick, Nischay Dhankhar, Gabor Fodor, and Sri Satish Ambati. H2O-Danube-1.8B Technical Report, 2024. URL https://arxiv.org/abs/2401.16818.

[FineWeb-Edu] Anton Lozhkov, Loubna Ben Allal, Leandro von Werra, and Thomas Wolf. FineWeb-Edu: the Finest Collection of Educational Content, 2024. URL https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu.

Dear Reviewer u4w2,

Thank you again for your review. Since the discussion period closes on November 18 (AoE), we wanted to check whether our responses address your concerns. In brief:

  • Novelty vs. ScaleCom and Radius: In SCAPE, the shard owners select the mask in parallel at every step, so the mask is never more than one step old. The mask acts on the optimizer state, so SCAPE also sparsifies the weight all-gather. Gradient-only methods like ScaleCom and Radius cannot reduce that all-gather, so under the distributed optimizer they cannot go below about one third of the dense bytes. SCAPE (d = 0.1) sends 12.4%.

  • DUO: DUO is now discussed in Appendix C. By our count, it sends about 75–90% of the dense volume. Against SDP4Bit, DUO's own baseline, SCAPE is level at 16–32 GPUs and 5–17% faster at 64–128 GPUs.

  • Runtime and time to matched quality: Fig. 6 shows SCAPE times against SDP4Bit and PowerSGD on 16–128 GB200 GPUs. SCAPE reaches its final validation loss in 28%, 43%, and 55% less time than dense AdamS needs to reach the same loss on 32, 64, and 128 GPUs.

  • Quality at 1.8B and the BoolQ drop: Llama-1.8B is now fully pretrained on 73.4B tokens. It ends within 0.013 of dense AdamS in validation loss, with seven of eight downstream tasks within one standard error. Retraining Llama-500M without data reuse removes the BoolQ drop (60.92 vs. 59.39). We train each configuration once, and the tables now report standard errors.

If these address your concerns, we would be grateful if you would consider updating your rating. If anything is still unclear, we are happy to clarify before the discussion closes.

Best regards, Authors of Submission 26608

Comments on the paper

1 comment

Dear Area Chair,

We are looking for your help with submission 26608. We have replied to all three reviewers, but none has responded so far.

For convenience, the main changes in the revision are:

  • Quality at scale, without data reuse (u4w2, EDPS): We fully pretrained Llama-1.8B on 73.4B FineWeb-Edu tokens and retrained Llama-500M on 41.9B tokens. At 90% sparsity, SCAPE ends within 0.013 (1.8B) and 0.026 (500M) of dense AdamS in validation loss, and the earlier BoolQ drop does not reproduce (Tables 1, 2 and 7; Fig. 5).

  • Runtime against compression baselines (u4w2, EDPS): Fig. 6 reports step time, scaling and peak memory against SDP4Bit (the authors' implementation) and PowerSGD on 16–128 GB200 GPUs. For Llama-8B on 128 GPUs, SCAPE is 4.09x faster than dense AdamS, versus 3.73x for SDP4Bit and 1.74x for PowerSGD, while sending about half of SDP4Bit's bytes.

  • Measurement details and cost (EDPS): We added the profiling method, collective dtypes, bucket sizes and achieved throughput (Appendix A, Table 3). Against an overlapped dense baseline, SCAPE is still 3.68x faster (Fig. 9). SCAPE sends 12.4% of the dense bytes per step and uses 13% more peak memory than dense AdamS.

  • Related work and scope (u4w2): DUO and ScaleCom are now discussed in Appendix C. Claims about quality and speedup are now stated for 90% sparsity, and 99% is presented as a limit test.

  • Theory and ablations (U8YQ): Remark F.12 shows how the analysis carries over to the delayed mask. The error-feedback ablation is in Sec. 6.3, and we added a fresh-vs-delayed mask ablation at 99% sparsity.

Thank you again for your time, and thank you for handling our submission.

Best regards,

Authors of Submission 26608

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.