- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper observes that the weights, activations, and gradients of LLMs can be approximated using GGD. Based on this, three methods are proposed, i.e., GG Init, ACT, and GCT. By minimizing a rate-based regularization term derived from Exponential-Golomb coding, these methods encourage the model to generate weights, activations, and gradients that are more compressible during training. Experiments across various models and tasks demonstrate that these methods reduce estimated storage and communication costs while largely maintaining model performance.
Critical strengths
-
Investigating weights, activations, and gradients simultaneously from the perspective of a unified statistical distribution is a valuable research direction.
-
GG Init, ACT, and GCT cover initialization, forward propagation, and backward propagation, resulting in a comprehensive overall framework.
-
The experiments encompass various model scales and tasks, demonstrating the potential for integration with quantization, sparsification, and entropy coding.
Critical weaknesses
-
BackSlash’s primary innovation lies in extending similar bitrate constraints to initialization, activation, and gradients. However, it lacks sufficient comparison with existing training-phase compression methods.
-
GCT’s second-order derivative approximation is based on SGD update dynamics, yet the experiments employ AdamW, creating a clear inconsistency between the theoretical derivation and the actual implementation.
-
The paper primarily reports estimated code lengths and communication volumes without fully implementing or evaluating actual encoding/decoding and distributed communication. Consequently, claims regarding end-to-end acceleration and hardware efficiency lack adequate support.
-
Experimental validation remains insufficient. Specifically, combined experiments involving GG Init, ACT, and GCT are missing, the mechanism by which GG Init affects pre-trained models is unclear, and most results lack statistics based on multiple random seeds or standard generation metrics.
Questions for authors
Please refer to the Critical Weaknesses part.
2 comments
Response (2/2)
Question #3: The paper mainly reports estimated code lengths and communication volumes without fully implementing or evaluating actual encoding/decoding and distributed communication. Consequently, claims regarding end-to-end acceleration and hardware efficiency lack adequate support.
Response: In pipeline/data-parallel distributed training of BERT with ACT and GCT, we actually encoded and transmitted communication tensors using the EG coding framework, and measured them over a conventional network of approximately 5 MB/s. The table below reports per-epoch communication volume, training time, encoding time, communication time, and end-to-end throughput. The results show that when network bandwidth is limited, communication time far exceeds training time, while EG encoding takes only 0.016–0.116 s, negligible relative to communication time. With only small increases in training time, ACT and GCT substantially reduce communication time, achieving throughput improvements of 1.63× and 1.75×, respectively.
| Method | Communication Volume | Training Time | Encoding Time | Communication Time | Throughput |
|---|---|---|---|---|---|
| ACT-Unconstrained (baseline) | 17.23GB | 139s | 0.026s | 3516s | 6.84 sample/s |
| ACT-Constrained | 10.29GB | 140s (+0.9%) | 0.016s (−38.5%) | 2101s (−40.2%) | 11.16 sample/s (1.63x) |
| GCT-Unconstrained (baseline) | 76.55GB | 217s | 0.116s | 15646s | 1.58 sample/s |
| GCT-Constrained | 43.10GB | 233s (+7.7%) | 0.066s (−43.1%) | 8851s (−43.4%) | 2.75 sample/s (1.75x) |
Question #4: Experimental validation remains insufficient. Specifically, combined experiments involving GG Init, ACT, and GCT are missing, the mechanism by which GG Init affects pretrained models is unclear, and most results lack statistics based on multiple random seeds or standard generation metrics.
Response: Combined experiments. We thank the reviewer for suggesting a combined experiment. These three methods act on different tensors at different stages: GG Init sets a static weight distribution at initialization for training from random initialization; ACT and GCT constrain dynamically generated forward activations and backward gradients, respectively, corresponding to communication in pipeline and data parallelism. Combining all three would simultaneously change the training starting point and apply two kinds of intermediate-quantity constraints, making their individual contributions difficult to separate. We therefore validate each method in the setting that corresponds to it: GG Init is compared with Kaiming initialization in training from random initialization, while ACT and GCT are compared with unconstrained training on pretrained models; results are reported across the model–task settings in Tables 3 and 5. Figure 6 in Appendix C illustrates the division of roles among the three methods, and Appendix A explains the compatibility of constrained training with conventional compression methods. This design makes it easier to assess whether each method is effective in its intended scenario.
Mechanism of GG Init. GG Init builds a low-entropy parameter distribution at initialization, and this low-entropy structure persists after training. Table 2 in Section 4.1 shows that, across shape parameters, the entropy at initialization closely matches the entropy after training; for ν = 0.1, for example, the values are 2.01 and 1.96. This indicates that GG Init suppresses the trained parameter bitrate through the initial parameter entropy distribution. Appendix B further analyzes theoretically how GG entropy varies with the shape parameter: entropy reaches its maximum at ν = 2, which corresponds to a Gaussian distribution, so Gaussian initialization is intrinsically unfavorable for weight compression.
Multiple random seeds. Because GG Init trains from random initialization, it has some variability; results across different seeds are reported in Table 6 and show a negligible seed effect. ACT and GCT perform constrained optimization on pretrained weights, where the optimization effect is relatively stable; experiments across models and datasets also corroborate this consistency. We further trained ACT and GCT with three seeds each on BERT/IMDB, and the results likewise show stability:
| Method | Accuracy | EG Code Length |
|---|---|---|
| ACT | 93.085% ± 0.140 pp | 5.971 ± 0.575 |
| GCT | 88.912% ± 1.315 pp | 4.327 ± 1.092 |
Generation metrics. We have added EM and F1 evaluation for Qwen4B model / WebQA dataset. We hope this offers a clearer presentation of the evaluation results for generation tasks.
| Method | Strategy | EM (%) | F1 (%) |
|---|---|---|---|
| ACT | Unconstrained | 20.52 | 37.23 |
| ACT | Constrained | 20.42 | 37.01 |
| GCT | Unconstrained | 20.23 | 36.93 |
| GCT | Constrained | 20.77 | 37.35 |
Response (1/2)
Question #1: BackSlash's main innovation lies in extending similar bitrate constraints to initialization, activations, and gradients. However, it lacks sufficient comparison with existing training-stage compression methods.
Response: We thank the reviewer for the valuable comment. This work is not a simple extension of BackSlash; it proposes a complete tensor-entropy constraint method based on constrained training. BackSlash previously identified the GGD properties of model weights; we extend this statistical perspective to weight initialization, activations, and gradients, and find that activations and gradients likewise exhibit low-shape-parameter GGD structure. Based on this statistical prior, we propose GG initialization and rate–distortion-constrained training methods for activations and gradients, substantially reducing bitrates.
Unlike post-training compression methods such as quantization and sparsification, ACT and GCT directly reshape tensor distributions during training to produce low-entropy representations that are easier to compress. As discussed in Appendix A, our approach is fundamentally a constrained-training algorithm: it reduces the entropy of weight, activation, and gradient distributions through training itself, thereby improving the intrinsic compressibility of the data. This distribution-level compression does not conflict with existing element-wise compression methods. The experiments in Appendix A show that GG Init can be combined with NF4 quantization and Wanda sparsification, ACT with LSQ quantization and ReLU-based sparsification, and GCT with QSGD quantization and Top-K sparsification; compared with unconstrained training, all of these combinations achieve higher compression ratios.
For comparison with existing training-stage compression methods, we compared ACT with the training-stage activation-entropy compression method CAT and with L1/L2 activation regularization under matched settings:
ACT achieves both the lowest code lengths and the highest accuracy.
Question #2: GCT's second-order derivative approximation is based on SGD update dynamics, yet the experiments use AdamW, creating a clear inconsistency between the theoretical derivation and the actual implementation.
Response: We thank the reviewer for pointing this out. We note that only one step in the GCT derivation depends on the SGD form: in Equation 13, the parameter displacement used for finite-difference curvature estimation is written as δw = −η·ĝ⁽ᵗ⁻¹⁾. The derivative of the rate term with respect to the weights (Equation 11) and the diagonal approximation (Equation 12) follow from the objective itself and are independent of the optimizer. GCT produces the equivalent gradient ĝ = g + λ_grad·ĥ; in the implementation, AdamW uses this gradient to update the parameters. The SGD-form update in Algorithm 2 is only for notational simplicity, and we will make this explicit in the final version.
Furthermore, under the approximation that momentum and weight decay are ignored, the actual AdamW displacement is δw = −η·ĝ⁽ᵗ⁻¹⁾/(√v̂ + ε_Adam), differing from the assumption in Equation 13 only by a positive per-coordinate scaling factor. Substituting this back into Equation 13 shows that the finite-difference curvature under AdamW equals the estimate in Equation 14 multiplied by (√v̂ + ε_Adam): the two have the same sign and differ only in magnitude across coordinates. GCT does not require an exact curvature value; it uses the curvature term to pass the objective of reducing gradient bitrate to the parameter update. The direction of the correction is determined by its sign, and its overall magnitude is calibrated by λ_grad, chosen empirically under AdamW. Equation 14 is therefore a sign-preserving local curvature proxy under AdamW.
Empirically, all results in the paper were obtained with AdamW: Figures 3(c)–(d) show that decaying GCT has a training loss largely matching conventional training while producing a substantially lower gradient bitrate, and the eight model–task settings in Tables 3 and 5 consistently confirm its effectiveness. Moreover, because AdamW already stores first- and second-order moments, the true displacement can be computed directly from the optimizer state, yielding an AdamW-consistent curvature estimate without additional GPU memory. We will provide this form in the revision.