acceptodds
Under review as a conference paper at ICLR 2027

Layer-wise Adaptive Gradient Compression with Generalized Error Estimation

Abstract

Reducing gradient communication is critical for large-scale data-parallel training. While layer-wise adaptive compression schemes can outperform uniform layer-wise ones, they typically rely on distance in gradient space as the error measure with a fixed measurement schedule. We propose Generalized Gradient Compression (GenComp), which generalizes layer-wise adaptive compression along three axes: 1) the space in which compression error is measured, on gradients or on latent activations obtained by simulating gradient updates and forward passes; 2) the error measure; 3) the frequency of measurement. We show that gradient- and activation-based error measurements, in tandem with error measures addressing information loss and different frequencies of measurement, induce distinct layer-wise compression dynamics, and can improve task performance and compression ratio. Across sparsification, quantization, and low-rank compression schemes applied to different model architectures, GenComp achieves significantly higher compression ratios than recent layer-wise adaptive schemes while retaining similar performance, which leads to reduced end-to-end training time for larger models.

Primary area:
optimization
Keywords:
Gradient compression, distributed training, error estimation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_2WyoOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes GenComp, an adaptive layer-wise gradient compression framework that generalizes how per-layer compression error is measured before selecting compression configurations. It varies three dimensions: the measurement space (gradients or activations after a simulated SGD step), the error metric (L2, cosine similarity, or KL divergence), and the measurement frequency (cyclic or one-shot). Experiments cover low-rank compression, top-k sparsification, and QSGD quantization on ResNet18/CIFAR-100, ResNet50/ImageNet, and Transformer-XL/WikiText-103. The paper also provides convergence guarantees for layer-wise contracting compressors and a bound on activation-to-output discrepancy.

Critical strengths
  1. The paper addresses a relevant and underexplored question: how compression error should be measured in layer-wise adaptive compression schemes.
  2. The framework is simple and general, encompassing uniform compression and approximately recovering L-GreCo as special cases.
  3. GenComp is compressor-agnostic and is evaluated across three major compression families: low-rank, sparsification, and quantization.
  4. Measuring error in the activation space after a simulated update provides a novel and intuitive proxy for a layer's functional change.
Critical weaknesses

W1.The central claims are stronger than the experimental evidence supports. The abstract and conclusion claim "significantly higher compression ratios ... while retaining similar performance," while Contribution 2 claims "full accuracy recovery." However, most compressed runs do not match the uncompressed baseline, and the reported differences are not supported by statistical significance tests.

W2. Comparisons at matched communication volumes are missing. The paper does not compare GenComp against uniform compression and other adaptive methods under the same cumulative communication budget. Such comparisons are essential to establish whether the proposed error measures improve the accuracy–communication trade-off. The paper should also report accuracy as a function of cumulative communicated bytes for all methods.

W3. The convergence theory does not cover the method as implemented.

  • Theorem 4 and Appendix B assume a single, fixed compression profile, whereas GenComp can update its profile during training.

  • The convergence bound does not depend on the error metric, measurement space, or error budget. It holds for any admissible compression profile and therefore does not distinguish GenComp from an arbitrary profile selection strategy.

  • The step-size condition force learning-rate decrease while the learning-rate scheduler used in the experiments violates this condition.

W4. Important practical limitations are not adequately addressed.

  • Communication budget: The method cannot directly target or reliably predict communication volume because its budget is expressed in error. The resulting compression ratios vary across error measures and substantially across random seeds.

  • Memory overhead: Memory costs are discussed only qualitatively. Neither an analytical memory-cost expression nor measured peak memory usage is provided.

  • Compression-cost modeling: The precomputed cost matrix (line 227) assumes fixed-size compressed representations, limiting applicability to methods such as threshold-based sparsification, whose communication costs vary with the gradient.

W5. Statistical reporting is insufficient.

  • Results are based on only three runs, without confidence intervals or statistical significance tests, despite the use of "significant" many times.
  • Some results report zero variability across three runs, such as the perplexity of 29.02 ± 0.00 in Table 2. These values require clarification.

W6. Notation, presentation, and mathematical consistency need improvement. The paper should adopt standard notation and remove symbol overloading.

Additional errors

  • The error-feedback update in B.1 will counts m_t twice.
  • The constants stated in Corollary B.16 is different than those derived in its proof.
  • The AI-use statement appears inconsistent: it states that AI was not used for "implementing methods" but was used for "creating and editing software code." This distinction should be clarified.
Questions for authors
  1. How do the best GenComp variants compare with uniform compression and other adaptive methods at the same cumulative communication volume?
  2. Which GenComp variant would the authors recommend as a default, and how should practitioners select a variant without exhaustively evaluating all combinations?
  3. How sensitive is the one-shot compression profile to the profiling batch size, the number of profiling batches, and the point in training at which profiling is performed?
  4. Can profile selection be formulated using a communication budget rather than an error budget, allowing practitioners to target a specific communication volume?
  5. What is the measured peak memory usage of GenComp compared with the baselines?
  6. How do the authors explain the reported standard deviations of ± 0.00 across three runs?
9 comments
AuthorsOpenReview ↗

Reviewer 2Wyo (1/5)

We thank the reviewer for their feedback and comments, and address their stated weaknesses and questions below. We have also updated the manuscript and highlighted all changes in blue.

W1. The central claims are stronger than the experimental evidence supports

We address this by providing Transformer-XL top-kk sparsification results at fixed training time and fixed communication budget (Tables R1 and R2) with new LEGACY and Accordion baseline results included. We also provide statistical significance tests for each method (Tables R3 and R4). We show that holding either of these fixed, most GenComp methods achieve significant improvements over baselines in perplexity, compression ratio, or both, in line with our claims.

Table R1: Transformer-XL top-kk sparsification fixed at 4 hours of training after warm-up

MethodIterationVal. perplexityCompression ratioCommunicated elements (×10¹⁰)
No compression4,713 ± 16841.90 ± 0.871.0 ± 0.071.27 ± 3.22
Uniform3,322 ± 14156.80 ± 1.905.0 ± 0.08.98 ± 0.54
Accordion3,772 ± 1949.44 ± 0.301.6 ± 0.032.77 ± 0.74
LEGACY-L9,342 ± 44635.79 ± 0.655.0 ± 0.031.78 ± 1.70
Gradient L2 norm, cyclic6,410 ± 13737.99 ± 0.2917.9 ± 0.65.81 ± 0.15
GenComp-G L2 norm, one-shot6,019 ± 9238.43 ± 0.4116.7 ± 0.35.77 ± 0.01
GenComp-G KLDiv, one-shot10,597 ± 20252.19 ± 6.24450.1 ± 1.00.41 ± 0.01
GenComp-G KLDiv, cyclic10,664 ± 32349.49 ± 0.69450.3 ± 1.10.41 ± 0.01
GenComp-A CosSim, one-shot9,060 ± 4733.81 ± 0.1461.7 ± 2.92.51 ± 0.11
GenComp-A CosSim, cyclic8,874 ± 28834.71 ± 0.1960.0 ± 6.82.54 ± 0.23
GenComp-A KLDiv, one-shot9,969 ± 34935.96 ± 0.14192.1 ± 37.10.92 ± 0.17
GenComp-A KLDiv, cyclic9,474 ± 17138.82 ± 0.8886.3 ± 7.71.89 ± 0.15
GenComp-A CSDiv, one-shot10,436 ± 16955.17 ± 11.49446.9 ± 4.40.41 ± 0.00
GenComp-A CSDiv, cyclic9,745 ± 5149.72 ± 2.74100.1 ± 1.91.68 ± 0.04

Table R2: Transformer-XL top-kk sparsification fixed at 5 x 10⁹ communicated elements after warm-up

MethodIterationVal. perplexityCompression ratio
No compression‡1,500 ± 0101.78 ± 0.601.0 ± 0.0
Uniform‡1,500 ± 0117.14 ± 1.855.0 ± 0.0
Accordion‡1,500 ± 0117.49 ± 0.675.0 ± 0.0
LEGACY-L‡1,500 ± 0105.89 ± 1.145.0 ± 0.0
Gradient L2 norm, cyclic‡1,500 ± 0110.90 ± 1.8816.3 ± 0.2
GenComp-G L2 norm, one-shot‡1,500 ± 0110.84 ± 1.0016.0 ± 0.3
GenComp-G KLDiv, one-shot12,970 ± 1549.20 ± 6.63459.5 ± 0.6
GenComp-G KLDiv, cyclic12,968 ± 1046.32 ± 0.68459.5 ± 0.4
GenComp-A CosSim, one-shot2,506 ± 7277.39 ± 1.3657.8 ± 2.8
GenComp-A CosSim, cyclic2,346 ± 19479.92 ± 5.1551.7 ± 7.5
GenComp-A KLDiv, one-shot5,818 ± 96848.45 ± 5.15184.9 ± 37.2
GenComp-A KLDiv, cyclic6,073 ± 6648.93 ± 1.29194.8 ± 2.5
GenComp-A CSDiv, one-shot12,924 ± 8849.60 ± 6.89457.8 ± 3.4
GenComp-A CSDiv, cyclic6,698 ± 1264.52 ± 2.99218.8 ± 0.5

‡ Uniform top-kk: already at 1.93 × 10¹⁰ communicated elements at its first validation after warm-up (step 1500). Gradient L2 norm, cyclic: between 5.85 × 10⁹ and 5.99 × 10⁹ communicated elements. GenComp-G L2 norm, one-shot: between 5.95 × 10⁹ and 6.08 × 10⁹ communicated elements. Accordion: 1.93 × 10¹⁰ communicated elements. LEGACY-L: 1.90 × 10¹⁰ communicated elements. No compression: 9.60 × 10¹⁰ communicated elements.

The above results show Transformer-XL sparsification at either fixed training time or fixed communication budget. The following is a table of Welch’s t-tests with Holm correction for the different methods against the gradient L2 norm baseline, where the null hypothesis is Δ = 0 for each metric compared to the corresponding gradient L2 norm value.

AuthorsOpenReview ↗

Reviewer 2Wyo (2/5)

Table R3: Top-kk sparsification, fixed training time: each method against Gradient L2 norm, cyclic

MethodΔ Val. perplexitypp HolmΔ Compression ratiopp HolmΔ Communicated elements (×10¹⁰)pp Holm
No compression+3.910.0100.070-16.9<0.0010.004+65.46<0.0010.003
Uniform+18.820.0030.022-12.9<0.0010.005+3.170.0070.013
Accordion+11.45<0.001<0.001-16.3<0.0010.004+26.96<0.0010.001
LEGACY-L-2.190.0150.092-12.8<0.0010.005+25.970.0010.004
GenComp-G L2 norm, one-shot+0.450.3380.706-1.20.0550.055-0.040.6560.656
GenComp-G KLDiv, one-shot+14.200.0580.234+432.3<0.001<0.001-5.40<0.0010.002
GenComp-G KLDiv, cyclic+11.50<0.0010.003+432.4<0.001<0.001-5.40<0.0010.002
GenComp-A CosSim, one-shot-4.18<0.0010.003+43.90.0010.006-3.30<0.001<0.001
GenComp-A CosSim, cyclic-3.28<0.0010.003+42.10.0080.025-3.27<0.0010.001
GenComp-A KLDiv, one-shot-2.020.0020.018+174.30.0150.030-4.89<0.001<0.001
GenComp-A KLDiv, cyclic+0.840.2350.706+68.50.0040.020-3.92<0.001<0.001
GenComp-A CSDiv, one-shot+17.180.2810.706+429.00.0040.020-5.40<0.0010.002
GenComp-A CSDiv, cyclic+11.730.0170.092+82.2<0.001<0.001-4.13<0.0010.001

We find that at the same given point in training time, GenComp-A CosSim, one-shot + cyclic and GenComp-A KLDiv, one-shot all have significantly lower perplexities and lower communication.

Table R4: Top-kk sparsification: After 5 × 10⁹ communicated elements after warm-up: each method against Gradient L2 norm, cyclic‡

MethodΔ Val. perplexitypp HolmΔ Compression ratiopp Holm
No compression‡-9.120.0090.043-15.3<0.001<0.001
Accordion‡+6.590.0170.060-11.3<0.001<0.001
LEGACY-L‡-5.010.0250.060-11.2<0.001<0.001
Uniform top-k‡+6.240.0150.060-11.3<0.001<0.001
GenComp-G L2 norm, one-shot‡-0.060.9680.968-0.30.3080.308
GenComp-G KLDiv, one-shot-61.700.0020.017+443.3<0.001<0.001
GenComp-G KLDiv, cyclic-64.57<0.001<0.001+443.2<0.001<0.001
GenComp-A CosSim, one-shot-33.51<0.001<0.001+41.60.0010.006
GenComp-A CosSim, cyclic-30.980.0050.027+35.40.0140.043
GenComp-A KLDiv, one-shot-62.45<0.0010.007+168.70.0160.043
GenComp-A KLDiv, cyclic-61.97<0.001<0.001+178.5<0.001<0.001
GenComp-A CSDiv, one-shot-61.300.0020.017+441.5<0.001<0.001
GenComp-A CSDiv, cyclic-46.38<0.001<0.001+202.5<0.001<0.001

‡ Already past the budget at the first validation after warm-up (iteration 1500), so read there: No compression at 9.60 × 10¹⁰ communicated elements; Accordion at 1.93 × 10¹⁰; LEGACY-L at 1.90 × 10¹⁰; Uniform top-k at 1.93 × 10¹⁰; Gradient L2 norm, cyclic at 5.85 × 10⁹ to 5.99 × 10⁹; GenComp-G L2 norm, one-shot at 5.95 × 10⁹ to 6.08 × 10⁹.

At a fixed communication budget, all GenComp methods except one-shot GenComp-G L2 norm have significantly lower perplexities than the gradient L2 norm baseline. The tables are also added to the revised manuscript in App. E.1.

Our abstract and conclusion claim of "significantly higher compression ratios ... while retaining similar performance” specifically compare GenComp to “recent layer-wise adaptive schemes”, as stated in the abstract, which we show for both Transformer-XL as well as ResNet-18/50 experiments. We acknowledge the conclusion statement can reiterate this more clearly, and have updated it to say “significantly higher compression rates than the layer-wise adaptive scheme baselines”. Regarding the “full accuracy recovery” claim in Contribution 2, we clarify that we compare one-shot measurements to cyclic error measurements, not to uncompressed training at the same number of iterations, and have specified this in the revised manuscript. We thank the reviewer for pointing out these clarifications.

AuthorsOpenReview ↗

Reviewer 2Wyo (3/5)

W2/Q1. Comparisons at matched communication volumes are missing

Comparisons at a fixed communication budget are provided in the tables above. We also provide a table of perplexity as a function of cumulative communicated bytes for all methods (each element is communicated in FP32 precision, i.e., 4 bytes per element):

Table R5: Top-kk sparsification: validation perplexity after a number of communicated bytes

MethodPerplexity @ 4 GB (10⁹ elements)Perplexity @ 40 GB (10¹⁰ elements)Perplexity @ 400 GB (10¹¹ elements)Perplexity @ 4 TB (10¹² elements)
No compression‡‡99.96 ± 0.6037.27 ± 0.21
Accordion‡‡58.92 ± 0.05†
LEGACY-L‡‡54.22 ± 0.33†
Uniform top-k‡‡53.67 ± 0.13†
Gradient L2 norm, cyclic‡88.28 ± 0.5429.29 ± 0.05†
GenComp-G L2 norm, one-shot‡88.04 ± 0.5231.09 ± 0.13†
GenComp-G KLDiv, one-shot123.82 ± 1.81†††
GenComp-G KLDiv, cyclic125.88 ± 2.28†††
GenComp-A CosSim, one-shot‡52.39 ± 0.72††
GenComp-A CosSim, cyclic‡54.93 ± 4.28††
GenComp-A KLDiv, one-shot133.70 ± 2.74 (2/3)35.00 ± 2.26††
GenComp-A KLDiv, cyclic139.21 ± 5.3344.14 ± 0.72††
GenComp-A CSDiv, one-shot122.76 ± 4.87†††
GenComp-A CSDiv, cyclic126.71 ± 4.6256.62 ± 2.55††

‡ Already past this budget at the first validation after warm-up (iteration 1,500). † Training ended before this budget.

The table is also added to the revised manuscript in App. E.1.

W3. The convergence theory does not cover the method as implemented

We thank the reviewer for these points and address them in turn.

Theorems B.7 and B.9 were indeed stated for a fixed profile. However, the analysis extends directly to profiles that change during training, provided the profile used at each step is chosen from information available before that step, as is the case for both one-shot and cyclic GenComp. The recursions for E∥xt−x~t∥2\mathbb{E}\lVert x_t - \tilde x_t\rVert^2 only involve the operator in use at the current step, and their coefficients are nondecreasing in the contraction constant. Hence, both theorems hold with c(ℓ)c^{(\ell)} replaced by the largest contraction constant among the configurations available to layer ℓ\ell. This also covers the warm-up phase, as uncompressed steps correspond to the identity operator. We have added this extension to Remark B.11.

We agree that the bounds hold for any admissible profile and therefore do not distinguish GenComp from other selection strategies. This is by design: the convergence results guarantee that training with error feedback converges for whichever profile GenComp selects, whereas the choice of error measure is motivated by the discrepancy bounds in Appendix C and assessed empirically. We now state this explicitly in Section 5 and Remark B.11.

We note that the condition αt2/αt+12≤4/(4−1/cmax⁡)\alpha_t^2/\alpha_{t+1}^2 \leq 4/(4-1/c_{\max}) does not force the learning rate to decrease: since the right-hand side exceeds one, it holds whenever αt+1≥αt\alpha_{t+1}\geq\alpha_t. It only limits how quickly the learning rate may decrease, requiring αt+1≥αt1−1/(4cmax⁡)\alpha_{t+1}\geq\alpha_t\sqrt{1-1/(4c_{\max})}. Constant learning rates (as in Corollary B.10) and linear warm-up satisfy it. We agree that abrupt drops, such as those of the step-decay schedule in our ResNet-18 experiments, violate the condition, and we now state this in Remark B.11.

We also note that the analysis covers SGD with error feedback and does not extend to the momentum-based optimizers or to PowerSGD with a single power iteration used in some of our experiments; we state this limitation explicitly in Remark B.11.

W4/Q4/Q5. Important practical limitations are not adequately addressed

Communication budget

With the minimization objective stated as in Eq. 1, where the maximum error is set from a uniform reference compression profile, we do get an upper bound on communication volume since the resulting compression profile from layer-wise profile selection must compress at least as much as the uniform reference profile or more. Our reported variation in what communication volume is achieved across error measures and variation over random seeds for a single measure is indeed a strength of the paper, as it shows the impact of error measures that have not been studied before.

It is also fully possible to more closely target a specific communication volume by inverting the optimization objective in Eq. 1 so we instead minimize layer-wise error with a constraint on the maximum total size of the compressed gradients: min⁡c∑ℓ=1Lδ(ℓ,cℓ)  s.t.  ∑ℓ=1Lω(ℓ,cℓ)≤Ωmax⁡\min_{\mathbf{c}} \sum_{\ell=1}^L \delta(\ell, c_\ell) \;\text{s.t.}\; \sum_{\ell=1}^L \omega(\ell, c_\ell) \le \Omega_{\max} instead of min⁡c∑ℓ=1Lω(ℓ,cℓ)  s.t.  ∑ℓ=1Lδ(ℓ,cℓ)≤Emax⁡\min_{\mathbf{c}} \sum_{\ell=1}^L \omega(\ell, c_\ell) \;\text{s.t.}\; \sum_{\ell=1}^L \delta(\ell, c_\ell) \le \mathcal{E}_{\max}.

Reviewer_wqQaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents an extension of layer-wise adaptive gradient compression that studies the space in which compression error is measured, the discrepancy measure, and the frequency of measurement. One of its main contributions is estimating compression sensitivity through changes in layer activations after simulated compressed and uncompressed updates. The authors further claim that selecting compression settings early in training reduces profiling overhead while maintaining competitive performance.

Critical strengths
  1. Measuring the effect of compression on activations is well motivated and a natural idea to explore. Two gradient perturbations with similar Euclidean error can have different effects on a layer’s output. The proposed estimator incorporates the layer’s inputs and provides a practical way to account for this difference within an existing compression-allocation framework.

  2. The one-shot profiling result can be practically useful. The experiments show that repeatedly updating compression settings is not always necessary, and avoiding these updates can reduce overhead without substantially changing performance. Including one-shot gradient (L_2) also helps show that this benefit can be obtained with a simple estimator.

Critical weaknesses
  1. The experiments do not clearly separate better error estimation from accepting more quality degradation. Several large compression and runtime gains come with worse final performance. For example, the 38.6% runtime reduction in the Transformer-XL sparsification experiment increases perplexity from 29.02 to 47.95. On ImageNet, uncompressed training is both faster and more accurate than every reported compressed variant. Comparisons at matched quality are needed to support the broader efficiency claims.

  2. The theoretical claims need correction and a clearer connection to the implementation. Corollary C.3 incorrectly equates total variation with the maximum difference between individual class probabilities. The convergence proof also uses second-moment bounds that do not follow from the stated first-moment assumptions. Beyond these errors, the activation-discrepancy theorem concerns quantities and assumptions that do not directly match the pooled, finite-batch scores used by the algorithm.

Questions for authors
  1. At the same communication budget or final model quality, do activation-based measures outperform gradient-based measures, particularly one-shot gradient (L_2)? This comparison would clarify how much of the improvement comes from the proposed estimator rather than profiling frequency or more aggressive compression.

  2. Can the authors correct the total-variation argument and the moment assumptions, and state precisely which guarantees apply to the implemented algorithm? In particular, how do the theoretical quantities relate to pooled activations, the specified MMD estimator, and the optimizers used in the experiments?

  3. Do the reported compression ratios include warm-up communication? For ImageNet, eight uncompressed epochs out of 90 would limit the whole-run compression ratio to 11.25, even with no communication afterward, whereas Table 5 reports substantially larger values. The accounting window and treatment of warm-up traffic need clarification.

3 comments
AuthorsOpenReview ↗

Reviewer wqQa (1/2)

We thank the reviewer for their feedback and comments, and address their stated weaknesses and questions below. We have also updated the manuscript and highlighted all changes in blue.

W1/Q1. Comparison at matched quality

Table R1: Transformer-XL top-kk sparsification fixed at 4 hours of training after warm-up

MethodIterationVal. perplexityCompression ratioCommunicated elements (×10¹⁰)
No compression4,713 ± 16841.90 ± 0.871.0 ± 0.071.27 ± 3.22
Uniform3,322 ± 14156.80 ± 1.905.0 ± 0.08.98 ± 0.54
Accordion3,772 ± 1949.44 ± 0.301.6 ± 0.032.77 ± 0.74
LEGACY-L9,342 ± 44635.79 ± 0.655.0 ± 0.031.78 ± 1.70
Gradient L2 norm, cyclic6,410 ± 13737.99 ± 0.2917.9 ± 0.65.81 ± 0.15
GenComp-G L2 norm, one-shot6,019 ± 9238.43 ± 0.4116.7 ± 0.35.77 ± 0.01
GenComp-G KLDiv, one-shot10,597 ± 20252.19 ± 6.24450.1 ± 1.00.41 ± 0.01
GenComp-G KLDiv, cyclic10,664 ± 32349.49 ± 0.69450.3 ± 1.10.41 ± 0.01
GenComp-A CosSim, one-shot9,060 ± 4733.81 ± 0.1461.7 ± 2.92.51 ± 0.11
GenComp-A CosSim, cyclic8,874 ± 28834.71 ± 0.1960.0 ± 6.82.54 ± 0.23
GenComp-A KLDiv, one-shot9,969 ± 34935.96 ± 0.14192.1 ± 37.10.92 ± 0.17
GenComp-A KLDiv, cyclic9,474 ± 17138.82 ± 0.8886.3 ± 7.71.89 ± 0.15
GenComp-A CSDiv, one-shot10,436 ± 16955.17 ± 11.49446.9 ± 4.40.41 ± 0.00
GenComp-A CSDiv, cyclic9,745 ± 5149.72 ± 2.74100.1 ± 1.91.68 ± 0.04

Table R2: Transformer-XL top-kk sparsification fixed at 5 x 10⁹ communicated elements after warm-up

MethodIterationVal. perplexityCompression ratio
No compression‡1,500 ± 0101.78 ± 0.601.0 ± 0.0
Uniform‡1,500 ± 0117.14 ± 1.855.0 ± 0.0
Accordion‡1,500 ± 0117.49 ± 0.675.0 ± 0.0
LEGACY-L‡1,500 ± 0105.89 ± 1.145.0 ± 0.0
Gradient L2 norm, cyclic‡1,500 ± 0110.90 ± 1.8816.3 ± 0.2
GenComp-G L2 norm, one-shot‡1,500 ± 0110.84 ± 1.0016.0 ± 0.3
GenComp-G KLDiv, one-shot12,970 ± 1549.20 ± 6.63459.5 ± 0.6
GenComp-G KLDiv, cyclic12,968 ± 1046.32 ± 0.68459.5 ± 0.4
GenComp-A CosSim, one-shot2,506 ± 7277.39 ± 1.3657.8 ± 2.8
GenComp-A CosSim, cyclic2,346 ± 19479.92 ± 5.1551.7 ± 7.5
GenComp-A KLDiv, one-shot5,818 ± 96848.45 ± 5.15184.9 ± 37.2
GenComp-A KLDiv, cyclic6,073 ± 6648.93 ± 1.29194.8 ± 2.5
GenComp-A CSDiv, one-shot12,924 ± 8849.60 ± 6.89457.8 ± 3.4
GenComp-A CSDiv, cyclic6,698 ± 1264.52 ± 2.99218.8 ± 0.5

‡ Uniform top-kk: already at 1.93 × 10¹⁰ communicated elements at its first validation after warm-up (step 1500). Gradient L2 norm, cyclic: between 5.85 × 10⁹ and 5.99 × 10⁹ communicated elements. GenComp-G L2 norm, one-shot: between 5.95 × 10⁹ and 6.08 × 10⁹ communicated elements. Accordion: 1.93 × 10¹⁰ communicated elements. LEGACY-L: 1.90 × 10¹⁰ communicated elements. No compression: 9.60 × 10¹⁰ communicated elements.

The tables above show Transformer-XL sparsification perplexity at a) the same wall-clock training time, or b) the same communication budget (number of elements communicated), with new LEGACY and Accordion baseline results included. We find that one-shot and cyclic GenComp-A CosSim, and one-shot GenComp-A KLDiv, have lower perplexity given the same training time as the cyclic gradient L2 norm baseline, and that almost all GenComp variants vastly outperform the baselines at a held communication budget. The tables are also added to the revised manuscript in App. E.1.

For the ResNet-50/ImageNet experiments, we acknowledge the performance drop and training time increase compared to the uncompressed baseline, and emphasize that despite using the same public source code to train the uniform and gradient L2 norm baselines as the uncompressed baseline, those two suffered even more from this degradation than the best-performing GenComp methods did. As stated in the paper, we argue that the increased communication bound from training larger models like Transformer-XL has greater potential for timing gains from aggressive gradient compression than what ResNet-50 does, which may explain the large improvement in Transformer-XL timing gains over the other models.

AuthorsOpenReview ↗

Reviewer wqQa (2/2)

W2/Q2. Corrections and clarifications of theoretical claims

We thank the reviewer for pointing out an inaccuracy in Corollary C.3 - we have updated this now in the manuscript. By using the centered delta kernel, the output space MMD equals the L2-distance between marginal output distributions, and thus controls both the total variation distance and maximum difference up to a factor depending on the number of classes.

Regarding the convergence proof: second-moment assumptions were intended, as the proofs use squared norms. We have corrected Assumptions B.5 and B.6 (and Assumption 3 in the main text) accordingly. For deterministic operators such as top-kk sparsification, this is equivalent to the original assumption and to that of [1]; for randomized operators, it is strictly stronger.

Regarding scope: Theorems B.7 and B.9 (and Theorem 4) apply to distributed SGD with error feedback, unbiased stochastic gradients, and layer-wise contracting operators satisfying Assumptions B.5 and B.6, which includes deterministic operators such as top-kk sparsification and exact low-rank truncation, and randomized operators such as scaled QSGD insofar as they satisfy the second-moment versions of these assumptions. As shown in Remark B.11 of the revised manuscript, they also cover profiles that change during training, provided each profile is selected using only information from previous steps and the contraction constants of each layer are bounded from above; both hold for one-shot and cyclic GenComp. They do not cover momentum SGD or LAMB, nor PowerSGD with a single power iteration. Since the bounds hold for any admissible profile, they guarantee that the profiles GenComp selects lead to convergence in this setting, but do not justify a particular error measure.

Regarding the pooling layers applied to convolutional layers: while it is true that there is a discrepancy between the pooled activation score and the raw activations used in C.3, we restate from Sec. 4 that we do this deliberately for dimensionality reduction and faster calculations. However, for any other type of layer, such as transformer blocks in Transformer-XL experiments, we only apply flattening to that layer. In this case, the activation scores described in App. C are the same as the ones used for GenComp error estimation in practice. The specified MMD estimator in App. C is the same one defined in App. D.3. Since the bounds in App. C concern the discrepancy frozen at a time step, and the same optimizer is used to obtain activations for both the compressed and uncompressed model, Theorem C.1 and Corollary C.3 hold for any optimizer including the ones used in GenComp experiments.

Q3. Warm-up period

The reported compression ratios do not include the warm-up period following the setup in [2]. During warm-up, uncompressed training is performed with dense gradient communication at all layers, as done in [2].

[1] Alistarh, D., Hoefler, T., Johansson, M., Khirirat, S., Konstantinov, N., and Renggli, C. (2018) The Convergence of Sparsified Gradient Methods. NeurIPS.

[2] Alimohammadi, M., Markov, I., Frantar, E., Alistarh, D. (2024) L-GreCo: Layerwise-adaptive Gradient Compression For Efficient Data-parallel Deep Learning. MLSys.

Dear Reviewer wqQa,

Thanks again for your efforts and interest in our paper. As the deadline for the author-reviewer discussion phase is approaching, we would like to ask if our responses have effectively addressed the concerns you raised. Should any remain unresolved, please let us know, and we will promptly follow up.

Reviewer_ncpoOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates layer-wise adaptive gradient compression for data-parallel training and proposes Generalized Gradient Compression (GenComp), a framework that generalizes compression configuration selection along the space in which compression error is measured, the error measure, and the frequency of measurement. GenComp-G evaluates compression error directly in gradient space, and GenComp-A simulates parameter updates using uncompressed gradients and their compressed-decompressed counterparts, and compares the resulting layer activations. Dynamic programming is then used to select per-layer compression configurations subject to an error budget derived from a reference uniform compression configuration. Experimental results show that these choices can improve compression ratios and reduce end-to-end training time.

Critical strengths
  1. Clear research problem and strong motivation. As neural networks continue to grow in scale, improving training efficiency through parallel computation becomes increasingly important. Assessing perturbations caused by gradient compression through changes in layer activations following simulated parameter updates is a natural and meaningful research direction.
  2. Theoretical analysis and a clearly defined optimization framework. Under the stated assumptions, the paper analyzes the convergence properties of layer-wise gradient compression and discusses the relationship between layer-wise activation discrepancies and perturbations in the model’s output distribution, providing theoretical motivation for activation-based compression error estimation. It also uses dynamic programming to select per-layer compression configurations subject to a global compression error budget.
Critical weaknesses
  1. The comparative efficiency claims need tighter qualification and more complete evaluation. For example, in the Transformer-XL sparsification experiments in Table 2, the largest reported reduction in training time relative to the baseline (38.6%) is accompanied by a substantial degradation in model quality. This result therefore does not support the broad claim of retaining similar task performance. Also, The baseline comparisons could be more comprehensive. The main-text experiments do not include comparisons with Accordion or LEGACY, leaving a relatively limited set of baselines for assessing the proposed method’s comparative efficiency and practical benefits. Although the appendix includes results for these methods, their compression ratios and total training times are not reported. Comparing accuracy alone, without reporting the associated communication and runtime costs, provides limited evidence for the claimed efficiency benefits. A broader set of baselines, together with more complete reporting of these metrics, would strengthen the evaluation.

  2. There is a gap in the convergence analysis. For randomized compression, the transition from Eq. (3) in Lemma B.14 to the first set of inequalities on page 25 requires ES∥Zt∥2≤ξ2αt2∥Gt∥2\mathbb{E}_S\|Z_t\|^2 \le \xi^2\alpha_t^2\|G_t\|^2, whereas Assumption B.5 currently provides only a bound on ES∥Zt∥\mathbb{E}_S\|Z_t\|. Here, for brevity, ZtZ_t denotes the vector difference inside the final squared norm in Eq. (3), and Gt:=G(ℓ)(xt)G_t := G^{(\ell)}(x_t) denotes the averaged gradient for the layer under consideration. A first-moment bound does not, in general, imply the required second-moment bound. Similarly, Lemma B.13 requires a comparison between expected squared norms, while Assumption B.6 only states a comparison between expected norms. I suggest that the authors clarify whether second-moment assumptions are intended.

Questions for authors
  1. Regarding empirical performance, could the authors identify compression configurations that achieve both lower perplexity and shorter total training time than the cyclic gradient-L2L_2 baseline? If such configurations exist, reporting both metrics in a table would make the empirical contribution and practical benefits of the proposed method clearer.

  2. Could the authors report the total training times for Accordion and LEGACY? If comparisons with these methods are difficult to conduct under a comparable experimental setup, could the authors include other relevant baselines?

  3. Please revisit the gap in the convergence analysis identified above and clarify or correct the corresponding derivation. If stronger assumptions are needed, please state them explicitly and clarify which randomized compression operators satisfy them and how this affects the scope of Theorem 4.

3 comments
AuthorsOpenReview ↗

Reviewer ncpo (1/2)

We thank the reviewer for their feedback and comments, and address their stated weaknesses and questions below. We have also updated the manuscript and highlighted all changes in blue.

W1/Q1. GenComp variants with lower perplexity and shorter training time

Tables 1 and 2 in the paper show results for each GenComp method after a set number of iterations, so GenComp variants that converge faster over time may appear to have degraded performance compared to the gradient L2 norm baseline. To address the reviewer’s concern, we provide our Transformer-XL sparsification results but with values read at a given timestep (4hrs of training) rather than after 15k training iterations (we also include new Accordion and LEGACY baseline results in our table):

Table R1: Transformer-XL top-kk sparsification fixed at 4 hours of training after warm-up

MethodIterationVal. perplexityCompression ratioCommunicated elements (×10¹⁰)
No compression4,713 ± 16841.90 ± 0.871.0 ± 0.071.27 ± 3.22
Uniform3,322 ± 14156.80 ± 1.905.0 ± 0.08.98 ± 0.54
Accordion3,772 ± 1949.44 ± 0.301.6 ± 0.032.77 ± 0.74
LEGACY-L9,342 ± 44635.79 ± 0.655.0 ± 0.031.78 ± 1.70
Gradient L2 norm, cyclic6,410 ± 13737.99 ± 0.2917.9 ± 0.65.81 ± 0.15
GenComp-G L2 norm, one-shot6,019 ± 9238.43 ± 0.4116.7 ± 0.35.77 ± 0.01
GenComp-G KLDiv, one-shot10,597 ± 20252.19 ± 6.24450.1 ± 1.00.41 ± 0.01
GenComp-G KLDiv, cyclic10,664 ± 32349.49 ± 0.69450.3 ± 1.10.41 ± 0.01
GenComp-A CosSim, one-shot9,060 ± 4733.81 ± 0.1461.7 ± 2.92.51 ± 0.11
GenComp-A CosSim, cyclic8,874 ± 28834.71 ± 0.1960.0 ± 6.82.54 ± 0.23
GenComp-A KLDiv, one-shot9,969 ± 34935.96 ± 0.14192.1 ± 37.10.92 ± 0.17
GenComp-A KLDiv, cyclic9,474 ± 17138.82 ± 0.8886.3 ± 7.71.89 ± 0.15
GenComp-A CSDiv, one-shot10,436 ± 16955.17 ± 11.49446.9 ± 4.40.41 ± 0.00
GenComp-A CSDiv, cyclic9,745 ± 5149.72 ± 2.74100.1 ± 1.91.68 ± 0.04

Table R1 identifies one-shot and cyclic GenComp-A CosSim, and one-shot GenComp-A KLDiv, as having lower perplexity given the same training time as the cyclic gradient L2 norm baseline. For users interested in reaching a specific target perplexity, this might be of particular interest as they can use the aforementioned methods to train for more iterations and reach the target perplexity in less runtime. We have also added the number of communicated elements after warm-up, which is directly determined by the compression rate.

While this table answers Q1, another important question for performance in communication-bound settings is how the models compare under a constraint on transmitted data volume; i.e., for a set number of transmitted elements, what perplexity does each method reach? This tells us something about what performance we expect in even more communication-bound settings such as when upscaling to even more GPUs, where gradient communication becomes a bigger bottleneck. We show this in Table R2, where we read values after a set number of gradient elements have been communicated (5 x 10⁹ elements):

Table R2: Transformer-XL top-kk sparsification fixed at 5 x 10⁹ communicated elements after warm-up

MethodIterationVal. perplexityCompression ratio
No compression‡1,500 ± 0101.78 ± 0.601.0 ± 0.0
Uniform‡1,500 ± 0117.14 ± 1.855.0 ± 0.0
Accordion‡1,500 ± 0117.49 ± 0.675.0 ± 0.0
LEGACY-L‡1,500 ± 0105.89 ± 1.145.0 ± 0.0
Gradient L2 norm, cyclic‡1,500 ± 0110.90 ± 1.8816.3 ± 0.2
GenComp-G L2 norm, one-shot‡1,500 ± 0110.84 ± 1.0016.0 ± 0.3
GenComp-G KLDiv, one-shot12,970 ± 1549.20 ± 6.63459.5 ± 0.6
GenComp-G KLDiv, cyclic12,968 ± 1046.32 ± 0.68459.5 ± 0.4
GenComp-A CosSim, one-shot2,506 ± 7277.39 ± 1.3657.8 ± 2.8
GenComp-A CosSim, cyclic2,346 ± 19479.92 ± 5.1551.7 ± 7.5
GenComp-A KLDiv, one-shot5,818 ± 96848.45 ± 5.15184.9 ± 37.2
GenComp-A KLDiv, cyclic6,073 ± 6648.93 ± 1.29194.8 ± 2.5
GenComp-A CSDiv, one-shot12,924 ± 8849.60 ± 6.89457.8 ± 3.4
GenComp-A CSDiv, cyclic6,698 ± 1264.52 ± 2.99218.8 ± 0.5

‡ Uniform top-kk: already at 1.93 × 10¹⁰ communicated elements at its first validation after warm-up (step 1500). Gradient L2 norm, cyclic: between 5.85 × 10⁹ and 5.99 × 10⁹ communicated elements. GenComp-G L2 norm, one-shot: between 5.95 × 10⁹ and 6.08 × 10⁹ communicated elements. Accordion: 1.93 × 10¹⁰ communicated elements. LEGACY-L: 1.90 × 10¹⁰ communicated elements. No compression: 9.60 × 10¹⁰ communicated elements.

AuthorsOpenReview ↗

Reviewer ncpo (2/2)

Table R2 demonstrates that if a user’s goal is to minimize perplexity while training in a heavily communication-bound setting, cyclic GenComp-G KLDiv achieves the lowest perplexity for our given communication budget, with all GenComp methods except one-shot GenComp-G L2 norm clearly surpassing the baselines, including the result that was referred to in W1 (one-shot GenComp-A CSDiv). The tables are also added to the revised manuscript in App. E.1.

W1/Q2. Accordion and LEGACY baselines

We provide new results for Accordion and LEGACY compared to selected GenComp methods (Tables R1 and R2). To measure effect on a larger-scale model and more communication-bound training, we performed new experiments on Transformer-XL. We follow their papers and evaluate training with top-kk sparsification, the only scheme for which the Accordion and LEGACY papers provide experimental results and compression profile recommendations for language models. For Accordion, we use ℓlow=0.99,ℓhigh=0.02\ell_{low} = 0.99, \ell_{high} = 0.02, their recommended profiles for training LSTM (their only tested language model). LEGACY provides Transformer-XL results for several configurations - we selected the configuration for which they reported the lowest perplexity, namely LEGACY-L with S70L9.9S_{70}L_{9.9}. We emphasize that in contrast, GenComp automatically selects a profile, removing the need for extensive search for compression profiles which both other methods rely upon.

While Accordion adapts to critical learning regimes during training to avoid performance degradation during those regimes, it applies very conservative sparsification when a regime is detected, yielding a very low compression ratio and substantially slower training performance. LEGACY-L divides layers into groups by their sizes, and requires handpicking kk ratios for each group. Their selected kk ratios reaching the lowest perplexity in their results gave a much more conservative compression ratio than the GenComp methods, and had a higher perplexity than GenComp-A CosSim, both cyclic and one-shot variants, in Table R1.

Since we had to implement and configure Accordion and LEGACY for this rebuttal, their runs were run with new code and configuration with more CPU cores, which made them about 3.5× faster per iteration in uncompressed warm-up (which otherwise took the same time across methods), with a lasting effect also later in training. Their results therefore show much faster throughput than what one would expect had they been tested at the same configuration as other runs. Even still, the GenComp-A CosSim variants beat both Accordion and LEGACY while at a significant disadvantage.

W2/Q3. Convergence analysis clarifications

We thank the reviewer for identifying this. Second-moment assumptions were intended: the proofs of Lemmas B.14 and B.15 use squared norms, whereas Assumptions B.5 and B.6 were stated with first moments, which do not imply the required bounds. We have corrected both assumptions (and Assumption 3 in the main text) to bound expected squared norms, e.g., ES∥S(ℓ)(vˉt(ℓ))−1P∑p=1PS(ℓ)(vtp,(ℓ))∥2≤ξ2∥αtG(ℓ)(xt)∥2\mathbb{E}_S\lVert S^{(\ell)}(\bar v_t^{(\ell)}) - \frac{1}{P}\sum_{p=1}^P S^{(\ell)}(v_t^{p,(\ell)})\rVert^2 \leq \xi^2\lVert \alpha_t G^{(\ell)}(x_t)\rVert^2 for Assumption B.5. The proofs then go through as written, and all constants are unchanged.

For deterministic operators (top-kk sparsification, exact rank-rr truncation), the corrected Assumption B.5 is equivalent to the original one and to the assumption of Alistarh et al. (2018), so the scope of Theorem 4 is unchanged. For randomized operators (random-kk, scaled QSGD), the second-moment versions are strictly stronger. We do not claim that any particular randomized operator satisfies them a priori; like the assumption of Alistarh et al. [1], they are analytic assumptions along the iterates, and we now state this explicitly in the paper.

We have also made the conditioning in the proofs explicit and stated the unbiasedness of the stochastic gradients. None of these changes affects the theorem statements.

[1] Alistarh, D., Hoefler, T., Johansson, M., Khirirat, S., Konstantinov, N., and Renggli, C. (2018). The Convergence of Sparsified Gradient Methods. NeurIPS.

Dear Reviewer ncpo,

Thanks again for your efforts and interest in our paper. As the deadline for the author-reviewer discussion phase is approaching, we would like to ask if our responses have effectively addressed the concerns you raised. Should any remain unresolved, please let us know, and we will promptly follow up.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.