- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper proposes GenComp, an adaptive layer-wise gradient compression framework that generalizes how per-layer compression error is measured before selecting compression configurations. It varies three dimensions: the measurement space (gradients or activations after a simulated SGD step), the error metric (L2, cosine similarity, or KL divergence), and the measurement frequency (cyclic or one-shot). Experiments cover low-rank compression, top-k sparsification, and QSGD quantization on ResNet18/CIFAR-100, ResNet50/ImageNet, and Transformer-XL/WikiText-103. The paper also provides convergence guarantees for layer-wise contracting compressors and a bound on activation-to-output discrepancy.
Critical strengths
- The paper addresses a relevant and underexplored question: how compression error should be measured in layer-wise adaptive compression schemes.
- The framework is simple and general, encompassing uniform compression and approximately recovering L-GreCo as special cases.
- GenComp is compressor-agnostic and is evaluated across three major compression families: low-rank, sparsification, and quantization.
- Measuring error in the activation space after a simulated update provides a novel and intuitive proxy for a layer's functional change.
Critical weaknesses
W1.The central claims are stronger than the experimental evidence supports. The abstract and conclusion claim "significantly higher compression ratios ... while retaining similar performance," while Contribution 2 claims "full accuracy recovery." However, most compressed runs do not match the uncompressed baseline, and the reported differences are not supported by statistical significance tests.
W2. Comparisons at matched communication volumes are missing. The paper does not compare GenComp against uniform compression and other adaptive methods under the same cumulative communication budget. Such comparisons are essential to establish whether the proposed error measures improve the accuracy–communication trade-off. The paper should also report accuracy as a function of cumulative communicated bytes for all methods.
W3. The convergence theory does not cover the method as implemented.
-
Theorem 4 and Appendix B assume a single, fixed compression profile, whereas GenComp can update its profile during training.
-
The convergence bound does not depend on the error metric, measurement space, or error budget. It holds for any admissible compression profile and therefore does not distinguish GenComp from an arbitrary profile selection strategy.
-
The step-size condition force learning-rate decrease while the learning-rate scheduler used in the experiments violates this condition.
W4. Important practical limitations are not adequately addressed.
-
Communication budget: The method cannot directly target or reliably predict communication volume because its budget is expressed in error. The resulting compression ratios vary across error measures and substantially across random seeds.
-
Memory overhead: Memory costs are discussed only qualitatively. Neither an analytical memory-cost expression nor measured peak memory usage is provided.
-
Compression-cost modeling: The precomputed cost matrix (line 227) assumes fixed-size compressed representations, limiting applicability to methods such as threshold-based sparsification, whose communication costs vary with the gradient.
W5. Statistical reporting is insufficient.
- Results are based on only three runs, without confidence intervals or statistical significance tests, despite the use of "significant" many times.
- Some results report zero variability across three runs, such as the perplexity of 29.02 ± 0.00 in Table 2. These values require clarification.
W6. Notation, presentation, and mathematical consistency need improvement. The paper should adopt standard notation and remove symbol overloading.
Additional errors
- The error-feedback update in B.1 will counts m_t twice.
- The constants stated in Corollary B.16 is different than those derived in its proof.
- The AI-use statement appears inconsistent: it states that AI was not used for "implementing methods" but was used for "creating and editing software code." This distinction should be clarified.
Questions for authors
- How do the best GenComp variants compare with uniform compression and other adaptive methods at the same cumulative communication volume?
- Which GenComp variant would the authors recommend as a default, and how should practitioners select a variant without exhaustively evaluating all combinations?
- How sensitive is the one-shot compression profile to the profiling batch size, the number of profiling batches, and the point in training at which profiling is performed?
- Can profile selection be formulated using a communication budget rather than an error budget, allowing practitioners to target a specific communication volume?
- What is the measured peak memory usage of GenComp compared with the baselines?
- How do the authors explain the reported standard deviations of ± 0.00 across three runs?
9 comments
Reviewer 2Wyo (2/5)
Table R3: Top- sparsification, fixed training time: each method against Gradient L2 norm, cyclic
| Method | Δ Val. perplexity | p | p Holm | Δ Compression ratio | p | p Holm | Δ Communicated elements (×10¹⁰) | p | p Holm |
|---|---|---|---|---|---|---|---|---|---|
| No compression | +3.91 | 0.010 | 0.070 | -16.9 | <0.001 | 0.004 | +65.46 | <0.001 | 0.003 |
| Uniform | +18.82 | 0.003 | 0.022 | -12.9 | <0.001 | 0.005 | +3.17 | 0.007 | 0.013 |
| Accordion | +11.45 | <0.001 | <0.001 | -16.3 | <0.001 | 0.004 | +26.96 | <0.001 | 0.001 |
| LEGACY-L | -2.19 | 0.015 | 0.092 | -12.8 | <0.001 | 0.005 | +25.97 | 0.001 | 0.004 |
| GenComp-G L2 norm, one-shot | +0.45 | 0.338 | 0.706 | -1.2 | 0.055 | 0.055 | -0.04 | 0.656 | 0.656 |
| GenComp-G KLDiv, one-shot | +14.20 | 0.058 | 0.234 | +432.3 | <0.001 | <0.001 | -5.40 | <0.001 | 0.002 |
| GenComp-G KLDiv, cyclic | +11.50 | <0.001 | 0.003 | +432.4 | <0.001 | <0.001 | -5.40 | <0.001 | 0.002 |
| GenComp-A CosSim, one-shot | -4.18 | <0.001 | 0.003 | +43.9 | 0.001 | 0.006 | -3.30 | <0.001 | <0.001 |
| GenComp-A CosSim, cyclic | -3.28 | <0.001 | 0.003 | +42.1 | 0.008 | 0.025 | -3.27 | <0.001 | 0.001 |
| GenComp-A KLDiv, one-shot | -2.02 | 0.002 | 0.018 | +174.3 | 0.015 | 0.030 | -4.89 | <0.001 | <0.001 |
| GenComp-A KLDiv, cyclic | +0.84 | 0.235 | 0.706 | +68.5 | 0.004 | 0.020 | -3.92 | <0.001 | <0.001 |
| GenComp-A CSDiv, one-shot | +17.18 | 0.281 | 0.706 | +429.0 | 0.004 | 0.020 | -5.40 | <0.001 | 0.002 |
| GenComp-A CSDiv, cyclic | +11.73 | 0.017 | 0.092 | +82.2 | <0.001 | <0.001 | -4.13 | <0.001 | 0.001 |
We find that at the same given point in training time, GenComp-A CosSim, one-shot + cyclic and GenComp-A KLDiv, one-shot all have significantly lower perplexities and lower communication.
Table R4: Top- sparsification: After 5 × 10⁹ communicated elements after warm-up: each method against Gradient L2 norm, cyclic‡
| Method | Δ Val. perplexity | p | p Holm | Δ Compression ratio | p | p Holm |
|---|---|---|---|---|---|---|
| No compression‡ | -9.12 | 0.009 | 0.043 | -15.3 | <0.001 | <0.001 |
| Accordion‡ | +6.59 | 0.017 | 0.060 | -11.3 | <0.001 | <0.001 |
| LEGACY-L‡ | -5.01 | 0.025 | 0.060 | -11.2 | <0.001 | <0.001 |
| Uniform top-k‡ | +6.24 | 0.015 | 0.060 | -11.3 | <0.001 | <0.001 |
| GenComp-G L2 norm, one-shot‡ | -0.06 | 0.968 | 0.968 | -0.3 | 0.308 | 0.308 |
| GenComp-G KLDiv, one-shot | -61.70 | 0.002 | 0.017 | +443.3 | <0.001 | <0.001 |
| GenComp-G KLDiv, cyclic | -64.57 | <0.001 | <0.001 | +443.2 | <0.001 | <0.001 |
| GenComp-A CosSim, one-shot | -33.51 | <0.001 | <0.001 | +41.6 | 0.001 | 0.006 |
| GenComp-A CosSim, cyclic | -30.98 | 0.005 | 0.027 | +35.4 | 0.014 | 0.043 |
| GenComp-A KLDiv, one-shot | -62.45 | <0.001 | 0.007 | +168.7 | 0.016 | 0.043 |
| GenComp-A KLDiv, cyclic | -61.97 | <0.001 | <0.001 | +178.5 | <0.001 | <0.001 |
| GenComp-A CSDiv, one-shot | -61.30 | 0.002 | 0.017 | +441.5 | <0.001 | <0.001 |
| GenComp-A CSDiv, cyclic | -46.38 | <0.001 | <0.001 | +202.5 | <0.001 | <0.001 |
‡ Already past the budget at the first validation after warm-up (iteration 1500), so read there: No compression at 9.60 × 10¹⁰ communicated elements; Accordion at 1.93 × 10¹⁰; LEGACY-L at 1.90 × 10¹⁰; Uniform top-k at 1.93 × 10¹⁰; Gradient L2 norm, cyclic at 5.85 × 10⁹ to 5.99 × 10⁹; GenComp-G L2 norm, one-shot at 5.95 × 10⁹ to 6.08 × 10⁹.
At a fixed communication budget, all GenComp methods except one-shot GenComp-G L2 norm have significantly lower perplexities than the gradient L2 norm baseline. The tables are also added to the revised manuscript in App. E.1.
Our abstract and conclusion claim of "significantly higher compression ratios ... while retaining similar performance” specifically compare GenComp to “recent layer-wise adaptive schemes”, as stated in the abstract, which we show for both Transformer-XL as well as ResNet-18/50 experiments. We acknowledge the conclusion statement can reiterate this more clearly, and have updated it to say “significantly higher compression rates than the layer-wise adaptive scheme baselines”. Regarding the “full accuracy recovery” claim in Contribution 2, we clarify that we compare one-shot measurements to cyclic error measurements, not to uncompressed training at the same number of iterations, and have specified this in the revised manuscript. We thank the reviewer for pointing out these clarifications.
Reviewer 2Wyo (3/5)
W2/Q1. Comparisons at matched communication volumes are missing
Comparisons at a fixed communication budget are provided in the tables above. We also provide a table of perplexity as a function of cumulative communicated bytes for all methods (each element is communicated in FP32 precision, i.e., 4 bytes per element):
Table R5: Top- sparsification: validation perplexity after a number of communicated bytes
| Method | Perplexity @ 4 GB (10⁹ elements) | Perplexity @ 40 GB (10¹⁰ elements) | Perplexity @ 400 GB (10¹¹ elements) | Perplexity @ 4 TB (10¹² elements) |
|---|---|---|---|---|
| No compression | ‡ | ‡ | 99.96 ± 0.60 | 37.27 ± 0.21 |
| Accordion | ‡ | ‡ | 58.92 ± 0.05 | † |
| LEGACY-L | ‡ | ‡ | 54.22 ± 0.33 | † |
| Uniform top-k | ‡ | ‡ | 53.67 ± 0.13 | † |
| Gradient L2 norm, cyclic | ‡ | 88.28 ± 0.54 | 29.29 ± 0.05 | † |
| GenComp-G L2 norm, one-shot | ‡ | 88.04 ± 0.52 | 31.09 ± 0.13 | † |
| GenComp-G KLDiv, one-shot | 123.82 ± 1.81 | † | † | † |
| GenComp-G KLDiv, cyclic | 125.88 ± 2.28 | † | † | † |
| GenComp-A CosSim, one-shot | ‡ | 52.39 ± 0.72 | † | † |
| GenComp-A CosSim, cyclic | ‡ | 54.93 ± 4.28 | † | † |
| GenComp-A KLDiv, one-shot | 133.70 ± 2.74 (2/3) | 35.00 ± 2.26 | † | † |
| GenComp-A KLDiv, cyclic | 139.21 ± 5.33 | 44.14 ± 0.72 | † | † |
| GenComp-A CSDiv, one-shot | 122.76 ± 4.87 | † | † | † |
| GenComp-A CSDiv, cyclic | 126.71 ± 4.62 | 56.62 ± 2.55 | † | † |
‡ Already past this budget at the first validation after warm-up (iteration 1,500). † Training ended before this budget.
The table is also added to the revised manuscript in App. E.1.
W3. The convergence theory does not cover the method as implemented
We thank the reviewer for these points and address them in turn.
Theorems B.7 and B.9 were indeed stated for a fixed profile. However, the analysis extends directly to profiles that change during training, provided the profile used at each step is chosen from information available before that step, as is the case for both one-shot and cyclic GenComp. The recursions for only involve the operator in use at the current step, and their coefficients are nondecreasing in the contraction constant. Hence, both theorems hold with replaced by the largest contraction constant among the configurations available to layer . This also covers the warm-up phase, as uncompressed steps correspond to the identity operator. We have added this extension to Remark B.11.
We agree that the bounds hold for any admissible profile and therefore do not distinguish GenComp from other selection strategies. This is by design: the convergence results guarantee that training with error feedback converges for whichever profile GenComp selects, whereas the choice of error measure is motivated by the discrepancy bounds in Appendix C and assessed empirically. We now state this explicitly in Section 5 and Remark B.11.
We note that the condition does not force the learning rate to decrease: since the right-hand side exceeds one, it holds whenever . It only limits how quickly the learning rate may decrease, requiring . Constant learning rates (as in Corollary B.10) and linear warm-up satisfy it. We agree that abrupt drops, such as those of the step-decay schedule in our ResNet-18 experiments, violate the condition, and we now state this in Remark B.11.
We also note that the analysis covers SGD with error feedback and does not extend to the momentum-based optimizers or to PowerSGD with a single power iteration used in some of our experiments; we state this limitation explicitly in Remark B.11.
W4/Q4/Q5. Important practical limitations are not adequately addressed
Communication budget
With the minimization objective stated as in Eq. 1, where the maximum error is set from a uniform reference compression profile, we do get an upper bound on communication volume since the resulting compression profile from layer-wise profile selection must compress at least as much as the uniform reference profile or more. Our reported variation in what communication volume is achieved across error measures and variation over random seeds for a single measure is indeed a strength of the paper, as it shows the impact of error measures that have not been studied before.
It is also fully possible to more closely target a specific communication volume by inverting the optimization objective in Eq. 1 so we instead minimize layer-wise error with a constraint on the maximum total size of the compressed gradients: instead of .
Reviewer 2Wyo (1/5)
We thank the reviewer for their feedback and comments, and address their stated weaknesses and questions below. We have also updated the manuscript and highlighted all changes in blue.
W1. The central claims are stronger than the experimental evidence supports
We address this by providing Transformer-XL top-k sparsification results at fixed training time and fixed communication budget (Tables R1 and R2) with new LEGACY and Accordion baseline results included. We also provide statistical significance tests for each method (Tables R3 and R4). We show that holding either of these fixed, most GenComp methods achieve significant improvements over baselines in perplexity, compression ratio, or both, in line with our claims.
Table R1: Transformer-XL top-k sparsification fixed at 4 hours of training after warm-up
Table R2: Transformer-XL top-k sparsification fixed at 5 x 10⁹ communicated elements after warm-up
‡ Uniform top-k: already at 1.93 × 10¹⁰ communicated elements at its first validation after warm-up (step 1500). Gradient L2 norm, cyclic: between 5.85 × 10⁹ and 5.99 × 10⁹ communicated elements. GenComp-G L2 norm, one-shot: between 5.95 × 10⁹ and 6.08 × 10⁹ communicated elements. Accordion: 1.93 × 10¹⁰ communicated elements. LEGACY-L: 1.90 × 10¹⁰ communicated elements. No compression: 9.60 × 10¹⁰ communicated elements.
The above results show Transformer-XL sparsification at either fixed training time or fixed communication budget. The following is a table of Welch’s t-tests with Holm correction for the different methods against the gradient L2 norm baseline, where the null hypothesis is Δ = 0 for each metric compared to the corresponding gradient L2 norm value.