acceptodds
Under review as a conference paper at ICLR 2027

It Takes a Good Model to Train a Good Model: Generalized Gaussian Priors for Optimized LLMs

Abstract

While the rapid advancement of large language models (LLMs) drives increased scale, data transfer, and computational demands, the statistical structures of their weights, activations, and gradients—and their implications for initialization, training dynamics, and efficiency—remain largely unexplored. We empirically show that these quantities in LLMs are well modeled by generalized Gaussian (GG) distributions, and introduce a unified, end-to-end optimization framework grounded in this observation. Our contributions are threefold: (1) a GG-based initialization that aligns with trained model statistics, accelerating training and improving accuracy; (2) activation constrained training (ACT), which regularizes activation representations with GG constraints, reducing pipeline-parallel transmission overhead; and (3) gradient constrained training (GCT) that enforces low-bitrate gradient structures, enhancing communication efficiency in distributed training. Experiments across diverse architectures demonstrate consistently smaller, faster models with minimal communication overhead that match or surpass standard baselines. By anchoring LLM optimization in principled statistical modeling, this work advances efficient, scalable, and hardware-aware AI systems.

Primary area:
optimization
Keywords:
Model Compression, Generalized Gaussian Priors, Rate-Distortion Optimization, Large Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_B9b4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper observes that the weights, activations, and gradients of LLMs can be approximated using GGD. Based on this, three methods are proposed, i.e., GG Init, ACT, and GCT. By minimizing a rate-based regularization term derived from Exponential-Golomb coding, these methods encourage the model to generate weights, activations, and gradients that are more compressible during training. Experiments across various models and tasks demonstrate that these methods reduce estimated storage and communication costs while largely maintaining model performance.

Critical strengths
  • Investigating weights, activations, and gradients simultaneously from the perspective of a unified statistical distribution is a valuable research direction.

  • GG Init, ACT, and GCT cover initialization, forward propagation, and backward propagation, resulting in a comprehensive overall framework.

  • The experiments encompass various model scales and tasks, demonstrating the potential for integration with quantization, sparsification, and entropy coding.

Critical weaknesses
  • BackSlash’s primary innovation lies in extending similar bitrate constraints to initialization, activation, and gradients. However, it lacks sufficient comparison with existing training-phase compression methods.

  • GCT’s second-order derivative approximation is based on SGD update dynamics, yet the experiments employ AdamW, creating a clear inconsistency between the theoretical derivation and the actual implementation.

  • The paper primarily reports estimated code lengths and communication volumes without fully implementing or evaluating actual encoding/decoding and distributed communication. Consequently, claims regarding end-to-end acceleration and hardware efficiency lack adequate support.

  • Experimental validation remains insufficient. Specifically, combined experiments involving GG Init, ACT, and GCT are missing, the mechanism by which GG Init affects pre-trained models is unclear, and most results lack statistics based on multiple random seeds or standard generation metrics.

Questions for authors

Please refer to the Critical Weaknesses part.

2 comments
AuthorsOpenReview ↗

Response (1/2)

Question #1: BackSlash's main innovation lies in extending similar bitrate constraints to initialization, activations, and gradients. However, it lacks sufficient comparison with existing training-stage compression methods.

Response: We thank the reviewer for the valuable comment. This work is not a simple extension of BackSlash; it proposes a complete tensor-entropy constraint method based on constrained training. BackSlash previously identified the GGD properties of model weights; we extend this statistical perspective to weight initialization, activations, and gradients, and find that activations and gradients likewise exhibit low-shape-parameter GGD structure. Based on this statistical prior, we propose GG initialization and rate–distortion-constrained training methods for activations and gradients, substantially reducing bitrates.

Unlike post-training compression methods such as quantization and sparsification, ACT and GCT directly reshape tensor distributions during training to produce low-entropy representations that are easier to compress. As discussed in Appendix A, our approach is fundamentally a constrained-training algorithm: it reduces the entropy of weight, activation, and gradient distributions through training itself, thereby improving the intrinsic compressibility of the data. This distribution-level compression does not conflict with existing element-wise compression methods. The experiments in Appendix A show that GG Init can be combined with NF4 quantization and Wanda sparsification, ACT with LSQ quantization and ReLU-based sparsification, and GCT with QSGD quantization and Top-K sparsification; compared with unconstrained training, all of these combinations achieve higher compression ratios.

For comparison with existing training-stage compression methods, we compared ACT with the training-stage activation-entropy compression method CAT and with L1/L2 activation regularization under matched settings:

StrategyEGHMFLAcc
ACT (EG)4.314.0913.0092.97%
L17.556.1612.0092.59%
L210.207.5912.0092.31%
CAT5.674.8013.0092.32%

ACT achieves both the lowest code lengths and the highest accuracy.

Question #2: GCT's second-order derivative approximation is based on SGD update dynamics, yet the experiments use AdamW, creating a clear inconsistency between the theoretical derivation and the actual implementation.

Response: We thank the reviewer for pointing this out. We note that only one step in the GCT derivation depends on the SGD form: in Equation 13, the parameter displacement used for finite-difference curvature estimation is written as δw = −η·ĝ⁽ᵗ⁻¹⁾. The derivative of the rate term with respect to the weights (Equation 11) and the diagonal approximation (Equation 12) follow from the objective itself and are independent of the optimizer. GCT produces the equivalent gradient ĝ = g + λ_grad·ĥ; in the implementation, AdamW uses this gradient to update the parameters. The SGD-form update in Algorithm 2 is only for notational simplicity, and we will make this explicit in the final version.

Furthermore, under the approximation that momentum and weight decay are ignored, the actual AdamW displacement is δw = −η·ĝ⁽ᵗ⁻¹⁾/(√v̂ + ε_Adam), differing from the assumption in Equation 13 only by a positive per-coordinate scaling factor. Substituting this back into Equation 13 shows that the finite-difference curvature under AdamW equals the estimate in Equation 14 multiplied by (√v̂ + ε_Adam): the two have the same sign and differ only in magnitude across coordinates. GCT does not require an exact curvature value; it uses the curvature term to pass the objective of reducing gradient bitrate to the parameter update. The direction of the correction is determined by its sign, and its overall magnitude is calibrated by λ_grad, chosen empirically under AdamW. Equation 14 is therefore a sign-preserving local curvature proxy under AdamW.

Empirically, all results in the paper were obtained with AdamW: Figures 3(c)–(d) show that decaying GCT has a training loss largely matching conventional training while producing a substantially lower gradient bitrate, and the eight model–task settings in Tables 3 and 5 consistently confirm its effectiveness. Moreover, because AdamW already stores first- and second-order moments, the true displacement can be computed directly from the optimizer state, yielding an AdamW-consistent curvature estimate without additional GPU memory. We will provide this form in the revision.

AuthorsOpenReview ↗

Response (2/2)

Question #3: The paper mainly reports estimated code lengths and communication volumes without fully implementing or evaluating actual encoding/decoding and distributed communication. Consequently, claims regarding end-to-end acceleration and hardware efficiency lack adequate support.

Response: In pipeline/data-parallel distributed training of BERT with ACT and GCT, we actually encoded and transmitted communication tensors using the EG coding framework, and measured them over a conventional network of approximately 5 MB/s. The table below reports per-epoch communication volume, training time, encoding time, communication time, and end-to-end throughput. The results show that when network bandwidth is limited, communication time far exceeds training time, while EG encoding takes only 0.016–0.116 s, negligible relative to communication time. With only small increases in training time, ACT and GCT substantially reduce communication time, achieving throughput improvements of 1.63× and 1.75×, respectively.

MethodCommunication VolumeTraining TimeEncoding TimeCommunication TimeThroughput
ACT-Unconstrained (baseline)17.23GB139s0.026s3516s6.84 sample/s
ACT-Constrained10.29GB140s (+0.9%)0.016s (−38.5%)2101s (−40.2%)11.16 sample/s (1.63x)
GCT-Unconstrained (baseline)76.55GB217s0.116s15646s1.58 sample/s
GCT-Constrained43.10GB233s (+7.7%)0.066s (−43.1%)8851s (−43.4%)2.75 sample/s (1.75x)

Question #4: Experimental validation remains insufficient. Specifically, combined experiments involving GG Init, ACT, and GCT are missing, the mechanism by which GG Init affects pretrained models is unclear, and most results lack statistics based on multiple random seeds or standard generation metrics.

Response: Combined experiments. We thank the reviewer for suggesting a combined experiment. These three methods act on different tensors at different stages: GG Init sets a static weight distribution at initialization for training from random initialization; ACT and GCT constrain dynamically generated forward activations and backward gradients, respectively, corresponding to communication in pipeline and data parallelism. Combining all three would simultaneously change the training starting point and apply two kinds of intermediate-quantity constraints, making their individual contributions difficult to separate. We therefore validate each method in the setting that corresponds to it: GG Init is compared with Kaiming initialization in training from random initialization, while ACT and GCT are compared with unconstrained training on pretrained models; results are reported across the model–task settings in Tables 3 and 5. Figure 6 in Appendix C illustrates the division of roles among the three methods, and Appendix A explains the compatibility of constrained training with conventional compression methods. This design makes it easier to assess whether each method is effective in its intended scenario.

Mechanism of GG Init. GG Init builds a low-entropy parameter distribution at initialization, and this low-entropy structure persists after training. Table 2 in Section 4.1 shows that, across shape parameters, the entropy at initialization closely matches the entropy after training; for ν = 0.1, for example, the values are 2.01 and 1.96. This indicates that GG Init suppresses the trained parameter bitrate through the initial parameter entropy distribution. Appendix B further analyzes theoretically how GG entropy varies with the shape parameter: entropy reaches its maximum at ν = 2, which corresponds to a Gaussian distribution, so Gaussian initialization is intrinsically unfavorable for weight compression.

Multiple random seeds. Because GG Init trains from random initialization, it has some variability; results across different seeds are reported in Table 6 and show a negligible seed effect. ACT and GCT perform constrained optimization on pretrained weights, where the optimization effect is relatively stable; experiments across models and datasets also corroborate this consistency. We further trained ACT and GCT with three seeds each on BERT/IMDB, and the results likewise show stability:

MethodAccuracyEG Code Length
ACT93.085% ± 0.140 pp5.971 ± 0.575
GCT88.912% ± 1.315 pp4.327 ± 1.092

Generation metrics. We have added EM and F1 evaluation for Qwen4B model / WebQA dataset. We hope this offers a clearer presentation of the evaluation results for generation tasks.

MethodStrategyEM (%)F1 (%)
ACTUnconstrained20.5237.23
ACTConstrained20.4237.01
GCTUnconstrained20.2336.93
GCTConstrained20.7737.35
Reviewer_8auGOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper models LLM weights, activations, and gradients as generalized Gaussian (GG) distributions. It proposes GG-based weight initialization (GG Init), plus training objectives that penalize the Exp-Golomb code length of activations (ACT) and gradients (GCT) to reduce communication in distributed training. Across eight models, it reports large bitrate reductions with little accuracy loss.

Are there any shortcomings in the method itself?

Finally, I manually organized the review comments provided by AI, selected and supplemented the weaknesses section.

Critical strengths

1.Shaping tensors to be compressible during training, rather than compressing them afterward, is a reasonable direction for communication-bound distributed training.

2.The evaluation covers eight models and reports system metrics (time, memory, communication volume). It also shows that the methods can be combined with existing quantization and sparsification techniques.

Critical weaknesses
  1. It is unclear whether accuracy is measured under actual compression. In Algorithm 1, quantization appears only inside the rate term, and the forward pass seems to use full-precision activations. The paper never states whether the reported accuracies come from models whose activations and gradients are actually quantized and entropy-coded. If they do not, the central claim of "lower bitrate with minimal accuracy loss" is not demonstrated.

  2. Data reliability.

    GG Init on BERT/IMDB reaches 87.11% after 5 epochs in Table 6 but only 82.23% after 20 epochs in Tables 3 and 9, and the paper gives no explanation.

    All results outside Table 6 come from single runs, while several claimed gains are within plausible noise (25.18 vs. 25.30, 38.83 vs. 39.16).

    Table 4 shows that Student-t fits activations and gradients better than GGD on most metrics. This contradicts the claim that activations and gradients are even more GG-like than weights.

  3. The GCT derivation is heuristic. The update rests on a chain of approximations: a diagonal Hessian, a finite-difference estimate from the previous step, substituting ĝ for g, an added 1/η term, and an ε smoothing constant. Each approximation lacks justification or an ablation. The derivation assumes SGD, but all experiments use AdamW, and this mismatch is not discussed. The method also needs a decaying λ to avoid late-stage oscillation, and the paper provides no convergence analysis.

Questions for authors

Same as weakness

2 comments
AuthorsOpenReview ↗

Response (1/2)

Question #1: It is unclear whether accuracy is measured under actual compression. In Algorithm 1, quantization appears only in the rate term, and the forward pass seems to use full-precision activations. The paper never states whether the reported accuracies come from models whose activations and gradients are actually quantized and entropy-coded. If not, the central claim of "lower bitrate with minimal accuracy loss" is not demonstrated.

Response: During training, the weights in GG Init, the activations in ACT, and the gradients in GCT are all quantized before EG encoding and transmission; the reported accuracies are therefore obtained under actual compression. We will add this process to the main text and Algorithm 1 to make the quantization, encoding, and transmission pipeline explicit.

Question #2a: GG Init on BERT/IMDB reaches 87.11% after 5 epochs in Table 6, but only 82.23% after 20 epochs in Tables 3 and 9, and the paper gives no explanation.

Response: To allow the model to converge fully, Tables 3 and 9 use 20 epochs. GG Init trains from random parameters, and prolonged training on a small dataset is prone to overfitting: the model over-adapts to the training samples, resulting in a slight drop in test performance. The final accuracies in Tables 3 and 9 are therefore lower than the 5-epoch result in Table 6. We will add this explanation to the main text.

Question #2b: All results outside Table 6 come from single runs, while several claimed gains are within plausible noise (25.18 vs. 25.30, 38.83 vs. 39.16).

Response: We have also trained ACT and GCT multiple times. Compared with GG Init, which starts from random initialization, ACT and GCT begin from pretrained weights and perform constrained optimization, a relatively stable process; the original manuscript therefore reported variance for GG Init only and did not report separate variances for ACT/GCT. To demonstrate this further, we repeated ACT and GCT three times each on BERT/IMDB, with the following results:

MethodAccuracyEG Code Length
ACT93.085% ± 0.140 pp5.971 ± 0.575
GCT88.912% ± 1.315 pp4.327 ± 1.092

In addition, the two pairs of results mentioned by the reviewer come from GG Init on DeepSeek-7B and Gemma-2B. For these results, our claim is not an accuracy improvement but a substantial reduction in parameter bitrate at comparable accuracy: the EG code lengths decrease from 5.26 to 4.29 and from 5.50 to 3.55, respectively.

Question #2c: Table 4 shows that Student-t fits activations and gradients better than GGD on most metrics. This contradicts the claim that activations and gradients are closer to GGD than weights are.

Response: First, our claim of being "closer" is relative to the Gaussian assumption: generalized Gaussian distributions are more flexible than Gaussians, and the low shape parameters exhibited by activations and gradients (Figure 1) show that they deviate substantially from a Gaussian. Second, Student-t fits are highly unstable and far less stable than GGD. For example, on the Wasserstein-1 metric for weights, Student-t and GGD give 65.87 vs. 0.0148 and 1.8e16 vs. 0.0577. GGD is therefore more suitable as a unified distribution for describing model tensors such as weights, activations, and gradients.

AuthorsOpenReview ↗

Response (2/2)

Question #3: The GCT derivation is heuristic. Its update relies on a chain of approximations: a diagonal Hessian, finite-difference estimates from the previous step, substituting ĝ for g, adding a 1/η term, and an ε smoothing constant. Each approximation lacks justification or an ablation. The derivation assumes stochastic gradient descent (SGD), but all experiments use AdamW, and this mismatch is not discussed. The method also needs a decaying λ to avoid late-stage oscillation, and the paper provides no convergence analysis.

Response: We thank the reviewer for examining each approximation in the derivation. Every term in Equations 11–15 serves the same purpose: obtaining a local rate correction that can be computed in parallel per parameter without computing the full Hessian.

  • The diagonal approximation removes second-order couplings between parameters so that the correction can be computed independently per coordinate, as in Equation 12.
  • The finite difference estimates diagonal curvature from the change in gradients between two adjacent steps; it requires storing only the previous step's equivalent gradient and no additional second-order backpropagation, as in Equation 13.
  • Substituting ĝ for g presupposes a small λ_grad, so the two are numerically close and the original gradient need not be stored separately; it also introduces a momentum-compensation term from the previous curvature that helps stabilize optimization.
  • The 1/η term comes from normalizing the displacement δw = η·ĝ⁽ᵗ⁻¹⁾, and ε is a smoothing constant that prevents numerical instability when the denominator approaches zero; together they form Equation 14.
  • The decaying λ_grad suppresses oscillations in late training.

For the difference between the SGD derivation and the AdamW implementation, please see our response to Reviewer 4, Question #2; for removing the second-order term or using a first-order soft-thresholding update, please see our responses to Reviewer 2, Question #3, and Follow-up Question #3.

We note that these approximations constitute a local surrogate objective. The paper does not provide a formal convergence proof for this update under AdamW, nor do we claim that it guarantees strict descent of the global objective. Empirically, Figure 3(c) shows late-stage oscillations when λ_grad is not decayed, whereas the loss curve of decaying GCT largely matches conventional training; Figure 3(d) shows that its gradient bitrate is lower and smoother. The eight model–task settings in Tables 3 and 5 further show that consistent rate reductions do not come at the cost of task performance. We will add derivations of these approximations in the revision.

Reviewer_NMX7OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper observes that LLM weights, activations, and gradients can be modeled by generalized Gaussian distributions (GGDs). Based on this observation, it proposes GGD-based weight initialization (GG Init) and extends BackSlash’s rate-constrained training to activations (ACT) and gradients (GCT). In specific, ACT and GCT penalize the corresponding Exp-Golomb code lengths during training to reduce communication costs in pipeline and data parallelism, respectively.

Critical strengths

This paper is well organized, and the observation is interesting.

Critical weaknesses

W1. Limited novelty over BackSlash (Wu et al., ICML 2025). The paper positions itself as an extension of BackSlash[1], but the extension is somewhat limited, and all three stated contributions are largely inherited from BackSlash. (i) GG structure: the paper reuse the GGD modeling of weights and the use of EG coding. What is added is fitting the same family to activations and gradients. (ii) ACT/GCT: both reuse BackSlash's objective J = D + λ·R, and the same EG/HM/FL evaluation protocol. The only change is an added penalization on the tensor. Beyond this, ACT adds a quantization step with an STE, and GCT adds an approximation for the Hessian computation. (iii) GG Init is the only component conceptually distinct from BackSlash, but its empirical support is weak (see W2). Overall, the paper reads as a set of follow-up experiments to BackSlash rather than independently substantiated contributions.

W2. The GG Init claim is not supported by the experiment. GG Init is presented as a weight initialization scheme for LLM training that "accelerates training", yet all experiments use pretrained checkpoints on downstream tasks, and Fig. 1 describes the setting as "full-parameter fine-tuning". If weights are loaded from the checkpoint, initialization is irrelevant. If instead the pretrained weights are discarded, then models are trained from scratch on small downstream datasets, which is neither LLM pretraining nor a realistic use case. The paper never states which is the case or which parameters are re-initialized, and no training curve or timing result for GG Init is reported, so the claimed acceleration is untested.

W3. The Hessian approximation used in GCT needs further justification. The method drops off-diagonal terms and estimates the diagonal from changes across training steps, but the paper does not evaluate the accuracy of this approximation or isolate its contribution to the results. A comparison with a simpler update using only the rate-gradient factor ρ\rho, or with soft-thresholding, would help establish whether the curvature correction provides enough benefit to justify its additional computation and memory.

W4. The practical benefit for distributed LLM training remains unclear. While the proposed methods reduce average code lengths and communication volume, these savings do not consistently translate into faster training. In particular, Table 8 shows increased training time for GCT despite lower communication volume. Evaluating throughput or time to reach the same model quality under different bandwidth conditions would help establish when the communication savings outweigh the additional overhead.

[1] Jun Wu, Jiangtao Wen, and Yuxing Han. “BackSlash: Rate Constrained Optimized Training of Large Language Models.”

Questions for authors
  1. For the GG Init experiments: are pretrained weights discarded? Which parameters does GG Init re-initialize? Why does the ν=2.0 baseline differ so much from the λ=0 baseline for the same model/task?
  2. Can the authors report any pretraining or convergence-speed result (perplexity vs. steps, or wall-clock) for GG Init to support "accelerating training"?
  3. For GCT: what is the effect of removing the second-order term (Eqs. 12–15) and using a plain first-order penalty on |g|? How does GCT compare with QSGD/Top-k/DGC at matched accuracy in actual transmitted bytes and wall-clock?
4 comments
AuthorsOpenReview ↗

Response (1/4)

Question #1: W1. Limited novelty over BackSlash (Wu et al., ICML 2025). The paper presents itself as an extension of BackSlash [1], but the extension is limited, and its three main contributions are largely inherited from BackSlash. (i) GG structure: the paper reuses GGD modeling of weights and EG coding; what is added is fitting the same family to activations and gradients. (ii) ACT/GCT: both reuse BackSlash's objective J = D + λ·R and the same EG/HM/FL evaluation protocol; the only difference is an added penalty on the tensor. Beyond this, ACT adds a quantization step using STE, and GCT adds an approximation of the Hessian computation. (iii) GG Init is the only conceptually distinct component, but its empirical support is weak (see W2). Overall, the paper reads as a series of follow-up experiments on BackSlash rather than independently substantiated contributions.

Response: This work is not a simple extension of BackSlash. BackSlash constrains model weights, i.e., the model itself, whereas ACT and GCT constrain intermediate quantities generated dynamically by data during training; this difference in the object of the constraint is fundamental. GG Init is the first low-entropy-oriented initialization method: it accounts for model compressibility at the initialization stage and obtains a low-entropy parameter representation from the outset. Our focus is instead on a unified framework for distribution modeling and constrained training. Specifically, this work differs from BackSlash in the following three respects.

  1. The GGD prior applies to different objects. BackSlash proposes a GGD prior only for weights, describing an intrinsic property of the model; we are the first to propose GGD priors for gradients and activations, describing dynamic properties of the data during training. Findings for model weights do not necessarily generalize to data-driven activations and gradients, so this extension of the prior has independent research significance.
  2. GG Init is an independent low-entropy initialization method. GG Init is the first low-entropy initialization method we propose; it is unrelated to BackSlash's constrained training, and the two share only the GGD prior for weights.
  3. ACT/GCT differ in both target and design. ACT and GCT compress intermediate representations in model training and inference, i.e., data features, rather than the fixed asset of model weights as in BackSlash; their processing is also more involved. ACT quantizes activations during the forward pass and makes the code length differentiable through STE; GCT constrains the rate of the gradients themselves and therefore requires differentiating this rate with respect to the weights, introducing second-order terms. Moreover, J = D + λR did not originate with BackSlash: it is the general form of rate–distortion optimization from Shannon's information theory and is widely used in image compression and many other fields. The similarity between ACT/GCT and BackSlash in objective form lies only in their shared use of rate–distortion optimization; their specific designs and target objects differ substantially.
AuthorsOpenReview ↗

Response (2/4)

Question #2: W2. The GG Init claim lacks experimental support. GG Init is described as a weight-initialization scheme for LLM training that "accelerates training", yet all experiments use pretrained checkpoints on downstream tasks, and Figure 1 describes the setting as "full-parameter fine-tuning". If weights are loaded from a checkpoint, initialization is irrelevant. If instead the pretrained weights are discarded, the model is trained from scratch on small downstream datasets, which is neither LLM pretraining nor a practical use case. The paper never states which is the case or which parameters are re-initialized, and reports no training curves or timing results for GG Init, so the claimed acceleration is unverified.

Response: Like Kaiming initialization, GG Init initializes all weight parameters in the model; the GG Init experiments therefore begin from random parameters and do not load pretrained weights. The "full-parameter fine-tuning" label in Figure 1 applies only to ACT and GCT, and we will revise it in the final version to distinguish clearly between the two experimental starting points.

Our experiments on small downstream datasets are intended to use datasets of different sizes to simulate how different amounts of injected information affect low-entropy initialization distributions, and to verify the advantages of GG Init over Kaiming initialization in model performance and parameter bitrate. We have also added a BERT pretraining experiment on WikiText-103. As shown below, GG Init retains a clear advantage over Kaiming initialization.

MethodEGHMFLAcc
GG Initialization4.573.5411.0042.27%
He Initialization5.454.4311.0040.73%

Regarding training acceleration, GG Init, ACT, and GCT all start from communication costs in multi-node, multi-GPU training; we therefore emphasize the reduction of communication waiting time in multi-node training, rather than claiming that the optimizer needs fewer training steps. We conducted multi-node training in a communication-constrained environment, and the results show that GG Init substantially reduces communication time. This is because the smaller average code length after encoding reduces the volume of data to be transmitted, while the computation required for encoding itself is inexpensive. When communication is the dominant bottleneck, transmitting less data means a lower communication bottleneck, and end-to-end communication is therefore accelerated. We will add this experiment and further clarify in the final version the scenario to which the training acceleration refers.

Question #3: W3. The Hessian approximation used in GCT requires further justification. The method drops off-diagonal terms and estimates the diagonal from changes across training steps, but the paper does not evaluate the accuracy of this approximation or isolate its contribution to the results. A comparison with a simpler update using only the rate-gradient factor, or with soft-thresholding, would help establish whether the curvature correction provides enough benefit to justify its additional computation and memory.

Response: We thank the reviewer for the suggestion. We first note that the curvature term in GCT is not an added correction; it arises necessarily from differentiating the rate term. GCT constrains the rate R(G) of the gradient G, and G = ∂D/∂w is itself a function of the weights, so the derivative of R(G) with respect to the weights is the product of ∂R/∂G and the second derivative ∂²D/∂w² (Equation 11). If the second-order term is removed and only the rate-gradient factor ∂R/∂G is retained, or if soft-thresholding is applied to the gradient, the resulting update is no longer the gradient of J_grad with respect to the weights; it is merely an element-wise post-processing of the current gradient. It can compress that gradient before transmission, but cannot change the distribution of subsequent gradients through the parameter update. Such post-processing is conventional post-training compression, complementary to rather than a substitute for GCT; Appendix A discusses this distinction. The curvature term is precisely the channel through which the objective of reducing gradient bitrate is passed to the model parameters.

In terms of overhead, we use a diagonal approximation and finite-difference estimation (Equations 12–14), requiring only the previous step's equivalent gradient ĝ to be stored and avoiding computation of the full Hessian matrix. For BERT in Table 8, GCT increases per-epoch training time from 2 min 18 s to 2 min 32 s and peak GPU memory from 6.21 GB to 7.08 GB, while reducing gradient communication volume from 2.95 TB to 1.33 TB. In bandwidth-constrained distributed training, this additional overhead is offset by the communication savings; see our response to Question #4.

AuthorsOpenReview ↗

Response (3/4)

Question #4: W4. The practical benefit for distributed LLM training remains unclear. Although the proposed methods reduce average code length and communication volume, these savings do not necessarily translate into faster training. In particular, Table 8 shows increased training time for GCT despite lower communication volume. Evaluating throughput or time to reach the same model quality under different bandwidth conditions would help establish when communication savings outweigh additional overhead.

Response: Table 8 reports training computation time only and excludes communication time; it therefore captures only the computational overhead added by GCT, not the benefit of communication savings. To address this, we measured ACT and GCT in pipeline/data-parallel distributed training of BERT over a conventional network of approximately 5 MB/s, transmitting tensors using the EG coding framework. The table below reports per-epoch communication volume, training time, encoding time, communication time, and end-to-end throughput. As can be seen, when network bandwidth is limited, communication time far exceeds training time. With only small increases in training time, ACT and GCT substantially reduce communication time, achieving throughput improvements of 1.63× and 1.75×, respectively.

MethodCommunication VolumeTraining TimeEncoding TimeCommunication TimeThroughput
ACT-Unconstrained (baseline)17.23GB139s0.026s3516s6.84 sample/s
ACT-Constrained10.29GB140s (+0.9%)0.016s (−38.5%)2101s (−40.2%)11.16 sample/s (1.63x)
GCT-Unconstrained (baseline)76.55GB217s0.116s15646s1.58 sample/s
GCT-Constrained43.10GB233s (+7.7%)0.066s (−43.1%)8851s (−43.4%)2.75 sample/s (1.75x)

Follow-up Question #1: For the GG Init experiments: are pretrained weights discarded? Which parameters does GG Init re-initialize? Why does the ν = 2.0 baseline differ so much from the λ = 0 baseline for the same model/task?

Response: Yes, the GG Init experiments do not use pretrained weights. As an initialization method, GG Init randomly initializes all weight parameters in the model; we compare it with Kaiming initialization, which corresponds to ν = 2.0, to show that it yields a low-entropy, more compressible parameter distribution while preserving accuracy. The difference between the ν = 2.0 and λ = 0 baselines comes from their different training starting points: the GG Init experiments train from random initialization, whereas the ACT and GCT experiments start from pretrained weights, and therefore achieve higher accuracy. We will state these settings explicitly in the main text.

Follow-up Question #2: Can the authors report any pretraining or convergence-speed results for GG Init (perplexity vs. steps, or wall-clock time) to support "accelerating training"?

Response: The pretraining results are given in our response to Question #2: on BERT/WikiText-103, GG Init achieves 42.27% accuracy, higher than Kaiming initialization's 40.73%, and an EG code length of 4.57, lower than Kaiming initialization's 5.45. "Accelerating training" refers to reducing communication time in communication-constrained multi-node training; the reason is explained in our response to Question #2. We will add GG Init training curves and communication timings in the revision and use them to clarify the scenario to which this statement applies.

AuthorsOpenReview ↗

Response (4/4)

Follow-up Question #3: For GCT: what is the effect of removing the second-order term (Equations 12–15) and using a simple first-order penalty on |g|? How does GCT compare with QSGD/Top-k/DGC at matched accuracy in actual transmitted bytes and wall-clock time?

Response: For the effect of removing the second-order term, please see our response to Question #3: applying a first-order penalty directly to |g| can compress only the current gradient and cannot change the distribution of subsequent gradients through the parameter update, whereas the second-order term is how GCT achieves this.

Regarding the comparison with QSGD/Top-k/DGC, GCT does not conflict with these conventional quantization and sparsification methods; it is complementary. QSGD, Top-k, and DGC operate on gradients that have already been generated and are therefore element-wise post-training compression methods; GCT reshapes the gradient distribution during training so that the gradients themselves become easier to compress. The two act at different stages and can be combined. Appendix A analyzes the compatibility of constrained training with conventional compression methods and experimentally verifies the combination of GCT with QSGD quantization and the Top-K sparsification used by DGC: compared with unconstrained training, the combinations achieve higher compression ratios with almost no loss of accuracy.

References

[1] Jun Wu, Jiangtao Wen, and Yuxing Han. BackSlash: Rate-Constrained Optimized Training of Large Language Models.

Reviewer_sBNKOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies generalized Gaussian distributions in model weights, activations, and gradients, and proposes GG-based initialization, activation constrained training (ACT), and gradient constrained training (GCT). The methods aim to improve compressibility by reducing coding rates during training. Experiments across several transformer models show substantial code-length reductions, with generally modest changes in the reported task metrics.

Critical strengths
  • Optimizing the compressibility of activations and gradients during training is a useful direction, with potential applications to communication-efficient training.

  • The appendix includes comparisons with alternative distribution models, layer-wise fitting results, and initialization experiments over five random seeds.

  • The experiments examine compatibility with quantization and sparsification and report training-time and memory overhead alongside coding rates.

Critical weaknesses
  • The novelty of ACT over prior compression-aware training is insufficiently established. Baskin et al. [1] already optimize activation entropy through differentiable regularization, including on BERT. The paper should demonstrate what its EG code-length objective adds beyond existing activation-entropy objectives under matched settings.

  • The contribution of the GG prior to the proposed optimization is unclear. BackSlash [2] already studies GG weight modeling and rate-constrained training, while Chen et al. [3] use generalized normal models for gradient entropy coding. The ACT/GCT objectives do not explicitly use fitted GG shape parameters, leaving it unclear whether the benefits depend on GG modeling or simply on encouraging compressibility. Comparisons with alternative regularizers would help establish this distinction.

  • The curvature approximation underlying GCT is not adequately justified. Equations 13–14 assume that parameter displacement equals the negative learning rate times the modified gradient, which does not generally hold for AdamW as used in the experiments. Consecutive minibatch gradients also differ because of the sampled data. The paper should validate whether this approximation captures the intended curvature under the actual training setup.

  • The evidence does not sufficiently support the claimed training-efficiency benefits. The main experiments primarily demonstrate reductions in code length, while the supplementary timing results do not show consistent training speedups. The paper should establish whether these reductions translate into improved distributed throughput after accounting for regularization, encoding, decoding, and synchronization costs.

  • Next-token accuracy alone does not establish preservation of generation quality. The question-answering and translation experiments should also report task-level metrics on generated outputs, such as answer exact match/F1 and a suitable translation metric.

References

[1] CAT: Compression-Aware Training for Bandwidth Reduction

[2] BackSlash: Rate-Constrained Optimized Training of Large Language Models

[3] DNN Gradient Lossless Compression: Can GenNorm Be the Answer?

Questions for authors

See weaknesses.

3 comments
AuthorsOpenReview ↗

Response (1/3)

Question #1: The novelty of ACT over prior compression-aware training methods has not been sufficiently established. Baskin et al. [1] already optimized activation entropy through differentiable regularization, including on BERT. The paper should demonstrate what its EG code-length objective adds beyond existing activation-entropy objectives under matched settings.

Response: We thank the reviewer for pointing out CAT [1] as related work. CAT constrains activation entropy with a differentiable regularizer, but this regularizer is not guided by a distributional prior and does not correspond to the number of bits required for actual encoding; it is therefore not rate–distortion optimization in the strict sense. ACT instead starts from the observation that activations follow a generalized Gaussian distribution with a low shape parameter: zeroth-order EG codes are well suited to such sources [4], and their code length is differentiable. We therefore use the average EG code length directly as the rate term in the loss (Equations 6–9). The rate optimized during training is thus kept consistent with the number of bits actually transmitted, and task performance and the compression cost of activation representations are traded off within one rate–distortion objective. In other words, we aim not only to make activations easier to compress, but also to use the GG prior to bring the required number of coding bits directly into the training objective, so that the model learns a more bit-efficient representation while performing the task.

We compared the ACT and CAT objectives under the same model, data, training budget, and quantization settings; the results are given in the table in our response to Question #2. ACT achieves EG/HM code lengths of 4.31/4.09, lower than CAT's 5.67/4.80, and increases accuracy from 92.32% to 92.97%.

Question #2: The contribution of GG before the proposed optimization method is unclear. BackSlash [2] already studied GG weight modeling and rate-constrained training, while Chen et al. [3] used generalized normal models for gradient entropy coding. The ACT/GCT objectives do not explicitly use fitted GG shape parameters, so it is unclear whether their benefits depend on GG modeling or merely on encouraging compressibility. Comparisons with other regularizers would help clarify this point.

Response: BackSlash [2] provides a GGD prior only for model weights, which are static model assets. We study activations and gradients, which are generated dynamically by data during training. A distributional prior that holds for weights does not necessarily carry over to these data-driven intermediate quantities, and therefore requires separate verification (Figure 1). Chen et al. [3] note that generalized Gaussian distributions may be suitable for describing gradients, but they encode gradients with Huffman codes, which do not fully exploit the structure of low-shape-parameter generalized Gaussian sources. Wen and Villasenor [4] discuss this in detail and show that structured prefix codes such as EG codes are better suited to such sources. Moreover, [3] considers only the coding cost at the communication stage and does not use the properties of generalized Gaussian distributions during training to guide the gradient distribution and achieve further compression.

In our framework, the GG prior plays two roles. GG modeling characterizes the distributional structure of tensors and provides the statistical basis for their compressibility; the matched EG code turns coding cost into a quantity that can be optimized directly during training. Thus, although the GG shape parameter does not appear explicitly in the objective, it determines the choice of the rate function: low-shape-parameter GG sources are precisely the regime in which EG codes perform best [4].

To verify this, we compared ACT, which directly optimizes the EG rate, with L1/L2 activation regularization and CAT [1]. L1/L2 constrain the magnitude or energy of activations and do not correspond directly to the actual number of coding bits, whereas EG has an explicit bitrate interpretation and aligns directly with our rate–distortion objective. As shown below, EG achieves both the lowest EG/HM code lengths and the highest accuracy.

StrategyEGHMFLAcc
EG4.314.0913.0092.97%
L17.556.1612.0092.59%
L210.207.5912.0092.31%
CAT [1]5.674.8013.0092.32%
AuthorsOpenReview ↗

Response (2/3)

Question #3: The curvature approximation underlying GCT is not adequately justified. Equations 13–14 assume that the parameter displacement equals the negative learning rate times the modified gradient, which generally does not hold for AdamW as used in the experiments. Moreover, consecutive minibatch gradients differ because of the sampled data. The paper should verify whether this approximation captures the intended curvature under the actual training setup.

Response: We thank the reviewer for raising these two points. GCT's curvature estimate targets the current minibatch gradient rather than the exact curvature of the global objective. At each step, the quantity actually encoded and transmitted is the instantaneous gradient produced by the current minibatch; GCT therefore takes the rate of this gradient as its local optimization target. The diagonal approximation removes second-order couplings between parameters, while finite differences estimate local curvature, turning the otherwise intractable gradient-rate objective into a computable local surrogate. Accordingly, the update guides the instantaneous gradient toward a lower-rate structure rather than being conditioned on strict descent of the global objective, and the decaying λ_grad keeps the correction close to the task gradient.

Regarding the fact that consecutive minibatch gradients come from different sampled data, the 1/η term and the denominator ĝ⁽ᵗ⁻¹⁾ in Equation 14 cancel the common scaling of the displacement magnitude at each update, while ε prevents the denominator from becoming too small. This estimate is therefore used primarily to form a local rate-correction direction, rather than interpreting the difference between two gradients directly as curvature. Figures 3(c)–(d) show that decaying GCT has a loss curve largely consistent with conventional training while producing a lower and smoother gradient bitrate; the eight model–task settings in Tables 3 and 5 likewise show consistent rate reductions with comparable task performance. For the relationship between AdamW and Equations 13–14, please see our response to Reviewer 4, Question #2.

Question #4: The available evidence does not sufficiently support the claimed training-efficiency improvements. The main experiments primarily show reductions in code length, while the supplementary timing results do not show consistent training speedups. The paper should further verify whether these reductions in code length translate into improved distributed throughput after accounting for regularization, encoding, decoding, and synchronization costs.

Response: In pipeline/data-parallel distributed training of BERT, we transmitted tensors using the EG coding framework and measured per-epoch communication volume, training time, encoding time, communication time, and end-to-end throughput over a conventional network of approximately 5 MB/s; the results are shown below. The timing results in the supplementary material include training computation time only and exclude communication time, so they capture only the computational overhead of the regularizers and not the benefit of communication savings. In bandwidth-constrained distributed settings, communication time far exceeds training time and becomes the dominant bottleneck. The additional computation introduced by ACT and GCT increases training time by only 0.9% and 7.7%, respectively, while encoding takes only 0.016–0.116 s, which is negligible relative to communication time. At the same time, communication time decreases by 40.2% and 43.4%, and end-to-end throughput improves by 1.63× and 1.75×, respectively. Moreover, encoding and decoding with zeroth-order EG codes are element-wise operations with a high degree of hardware parallelism.

MethodCommunication VolumeTraining TimeEncoding TimeCommunication TimeThroughput
ACT-Unconstrained (baseline)17.23GB139s0.026s3516s6.84 sample/s
ACT-Constrained10.29GB140s (+0.9%)0.016s (−38.5%)2101s (−40.2%)11.16 sample/s (1.63x)
GCT-Unconstrained (baseline)76.55GB217s0.116s15646s1.58 sample/s
GCT-Constrained43.10GB233s (+7.7%)0.066s (−43.1%)8851s (−43.4%)2.75 sample/s (1.75x)
AuthorsOpenReview ↗

Response (3/3)

Question #5: Next-token accuracy alone does not establish preservation of generation quality. The question-answering and translation experiments should also report task-level metrics on generated outputs, such as answer exact match/F1 and a suitable translation metric.

Response: We thank the reviewer for the suggestion. Next-token accuracy measures the correctness of step-by-step prediction, whereas EM and F1 evaluate complete generated outputs; the two are complementary. We therefore additionally evaluated EM and F1 on free-generation outputs from Qwen3-4B on WebQuestions, as shown below.

MethodStrategyEM (%)F1 (%)
ACTUnconstrained20.5237.23
ACTConstrained20.4237.01
GCTUnconstrained20.2336.93
GCTConstrained20.7737.35

EM and F1 for both ACT and GCT are essentially unchanged from unconstrained training, indicating that activation and gradient constraints also preserve generation quality on generative tasks.

References

[1] CAT: Compression-Aware Training for Bandwidth Reduction.

[2] BackSlash: Rate-Constrained Optimized Training of Large Language Models.

[3] DNN Gradient Lossless Compression: Can GenNorm Be the Answer?

[4] J. Wen and J. D. Villasenor. Structured Prefix Codes for Quantized Low-Shape-Parameter Generalized Gaussian Sources. IEEE Transactions on Information Theory, 1999.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.