- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Generalized Residual Constrained Training (GRCT), a rate–distortion framework that jointly optimizes task performance and Transformer structure during end-to-end training. By modeling generalized residuals with generalized Gaussian statistics and penalizing their Exp-Golomb coding rates, GRCT adaptively suppresses and removes redundant layers, sublayers, or layer segments without requiring post-training pruning and recovery. Experiments across multiple LLMs and tasks show that GRCT preserves task performance while substantially reducing training time, memory usage, inference latency, and model size.
Critical strengths
This paper learns deployable compact models directly during training, thereby reducing both training and inference costs rather than only improving post-training efficiency. The extensive experiments, trajectory-aware efficiency metrics, and analysis of residual distributions and pruning patterns provide strong empirical support for the method’s effectiveness and adaptability across model scales and task types.
Critical weaknesses
-
The decision to delete the paper was not directly linked to the variation in validation loss. The task loss can constrain the structural contraction, but the low gating weighted encoding cost is not directly equivalent to low task importance. The paper has not fully explained why it is sufficient as a basis for permanent deletion. Is this judgment mainly based on empirical observation?
-
The paper permanently deletes the structure based on Formula 9 and Algorithm 1. However, the model actually incorporates the hidden state equation gr. Please explain the actual gating residual amplitude difference of each layer, as well as the variation in validation loss before and after deletion, to illustrate that the low pruning score may be due to predicting low deletion damage.
-
The changes in regular strength and pruning intensity in the paper have not answered whether EG residual regularization is superior to other methods under the same gating training budget and physical deletion strategy. It is necessary to prove that its residual cost and continuous deletion process bring effects, and cannot mainly rely on the broad positioning of joint optimization of training and structure.
-
The main text describes ShortGPT, LaCo, and SliceGPT as pruning baselines without recovery training, but Appendix C.1 indicates that all methods have undergone the same number of task training epochs. The text is not clear about whether these training are conducted before or after pruning, and whether the baseline starts from the same fine-tuning weights. It is recommended to provide additional explanations.
-
The table reports training and inference acceleration, but does not list the task accuracy of these accelerated models, nor does it list the total time taken for the pruning baseline from the initial weights to the final model.
-
The generation task is evaluated only by PPL, which cannot fully reflect the generation ability of the pruned model. It is recommended to supplement SQuAD's EM/F1, as well as the accuracy pass@1 on the code benchmark, to verify whether the model can still maintain performance under this high pruning rate.
-
In Section 5 of the paper, teacher forcing is used to explain the excessive deletion of Attention by GRCT-SL on SQuAD. However, the current basis is mainly the pruning distribution and the differences in training/testing losses. The current explainability is insufficient. It is recommended to adjust the number of Attention deletions under the same pruning budget, or to separately adjust the regular strength of Attention and FFN to observe the generation performance.
-
Please clarify whether the pruning rate in the cross-table results is uniformly defined. There is inconsistency in Tables 1, 2, and Figure 7, and the corresponding accuracies are not exactly the same. Please indicate whether these results come from different configurations, or different random seeds, or whether the differences in pruning rates should be verified.
Questions for authors
See Weakness.
2 comments
Response (2/2)
Concern #5: The table reports training and inference acceleration, but does not list the task accuracy of these accelerated models, nor does it list the total time taken for the pruning baseline from the initial weights to the final model.
Response: We have added task accuracy and total training time for Dense, GRCT, and GRCT-SL in Table 2. Taking LLaMA-3.2-1B on IMDB as an example, all five external pruning baselines start from the same Dense checkpoint, which takes 112.68 min for training. ShortGPT, LaCo, and SliceGPT then take an additional 7.57 s, 45.34 s, and 9.80 s for pruning, bringing their end-to-end times to 112.81, 113.44, and 112.84 min, respectively. LLM-Pruner takes an additional 23.03 min for pruning and recovery fine-tuning, and SkipGPT takes an additional 67.93 min for router training and recovery fine-tuning, bringing their totals to 135.71 and 180.61 min, respectively. We will report end-to-end times for all external pruning baselines in the revised manuscript so that readers can compare training cost and task accuracy together.
| Model | Method | Accuracy | Training Time (min) |
|---|---|---|---|
| LLaMA-3.2-1B | Dense | 95.38% | 112.68 |
| LLaMA-3.2-1B | GRCT-SL | 93.56% | 108.74 |
| LLaMA-3.2-1B | GRCT | 93.90% | 93.35 |
| Qwen3-4B | Dense | 95.74% | 460.37 |
| Qwen3-4B | GRCT-SL | 93.79% | 392.39 |
| Qwen3-4B | GRCT | 94.56% | 366.22 |
| LLaMA-3.1-8B | Dense | 95.37% | 704.88 |
| LLaMA-3.1-8B | GRCT-SL | 94.26% | 600.10 |
| LLaMA-3.1-8B | GRCT | 94.45% | 460.03 |
| Qwen3-14B | Dense | 96.34% | 1357.30 |
| Qwen3-14B | GRCT-SL | 94.26% | 1110.07 |
| Qwen3-14B | GRCT | 93.78% | 983.87 |
Concern #6: It is recommended to supplement SQuAD's EM/F1, as well as the accuracy pass@1 on the code benchmark, to verify whether the model can still maintain performance under this high pruning rate.
Response: We further report CodeAlpaca pass@1/Next Token Accuracy (NTA), and SQuAD EM/F1 in the table below to evaluate GRCT more comprehensively. The results show that GRCT achieves performance comparable to Dense across all metrics.
| Model | Method | CodeAlpaca (pass@1) | CodeAlpaca (NTA) | SQuAD (EM) | SQuAD (F1) |
|---|---|---|---|---|---|
| LLaMA-3.2-1B | Dense | 39.60% | 78.18% | 48.52% | 60.83% |
| LLaMA-3.2-1B | GRCT | 35.00% | 75.64% | 52.16% | 64.20% |
| Qwen3-4B | Dense | 54.40% | 81.33% | 67.64% | 79.21% |
| Qwen3-4B | GRCT | 48.00% | 77.99% | 69.61% | 80.73% |
| LLaMA-3.1-8B | Dense | 41.00% | 77.97% | 58.79% | 70.39% |
| LLaMA-3.1-8B | GRCT | 39.70% | 77.27% | 59.23% | 71.99% |
| Qwen3-14B | Dense | 32.00% | 77.12% | 67.12% | 80.33% |
| Qwen3-14B | GRCT | 35.30% | 78.19% | 63.22% | 75.62% |
Concern #7: It is recommended to adjust the number of Attention deletions under the same pruning budget, or to separately adjust the regular strength of Attention and FFN to observe the generation performance.
Response: Thank you for your such insightful suggestion. In LLaMA-3.2-1B / SQuAD experiments, we reduced the Attention constraint strength from 0.2 to 0.01, achieving a more balanced pruning allocation between Attention and FFN. Despite its higher pruning rate, GRCT-SL with the reduced Attention constraint achieved substantially lower PPL, outperforming Dense. These results indicate that adjusting the pruning allocation between Attention and FFN can substantially improve GRCT-SL’s performance.
| Method | Attention lambda | FFN lambda | Pruning Rate | PPL | Attention Pruned | FFN Pruned |
|---|---|---|---|---|---|---|
| Dense | 0 | 0 | 0.0% | 52.77 | 0 / 16 | 0 / 16 |
| GRCT-SL | 0.2 | 0.2 | 25.6% | 116.43 | 11 / 16 | 4 / 16 |
| GRCT-SL | 0.01 | 0.2 | 29.5% | 44.15 | 6 / 16 | 6 / 16 |
Concern #8: Please clarify whether the pruning rate in the cross-table results is uniformly defined. There is inconsistency in Tables 1, 2, and Figure 7, and the corresponding accuracies are not exactly the same. Please indicate whether these results come from different configurations, or different random seeds, or whether the differences in pruning rates should be verified.
Response: Only for Qwen3-14B / GRCT in Table 2, we slightly increased the constraint strength to λ=0.03 to narrow the pruning-rate gap between GRCT and GRCT-SL and improve their comparability of system efficiency. Apart from this adjustment, Tables 1 and 2 and Figure 7 use the same base training settings. Differences in the reported metrics arise from independent training runs, which converge to comparable levels. All parameter-pruning rates use the same metric, defined as the number of removed parameters divided by the total number of parameters before pruning, including embeddings and the LM head.
Response (1/2)
We sincerely appreciate your thorough review and thoughtful suggestions, which have helped us present our contributions more clearly and provide stronger experimental support. In this paper, we propose GRCT, a constrained training method that enables models to adaptively learn their structure based on task requirements. Thank you for recognizing GRCT’s training and inference efficiency, empirical support, and adaptability across models and tasks, as well as the value of our evaluation of structural evolution and residual analysis. Guided by your constructive feedback, we have refined the methodological explanations and added supporting experiments. We hope our responses below satisfactorily address your concerns.
Concern #1: The decision to delete the paper was not directly linked to the variation in validation loss. The task loss can constrain the structural contraction, but the low gating weighted encoding cost is not directly equivalent to low task importance. The paper has not fully explained why it is sufficient as a basis for permanent deletion. Is this judgment mainly based on empirical observation?
Response: A low residual rate actually reflects a sparse, low-magnitude residual distribution and indicates a negligible contribution to the forward hidden states and the model’s final predictions, making it a proxy for pruning. Meanwhile, GRCT continuously balances residual rate against task loss, allowing each layer’s residuals to converge smoothly and stably to a well-defined level, thereby providing a guarantee for permanent pruning based on residual rate. The table below shows the residual-RMS trajectories of retained layer L0 and pruned layer L12, with L12’s residual RMS converging to an extremely low level. We will include the trajectories of all layers in the revised manuscript.
Concern #2: Please explain the actual gating residual amplitude difference of each layer, as well as the variation in validation loss before and after deletion, to illustrate that the low pruning score may be due to predicting low deletion damage.
Response: Using training on LLaMA-3.2-1B / IMDB as an example, we report the residual RMS of each layer at its removal and the changes in loss and accuracy across the entire test set before and after removal. The results show that the residual magnitudes are already low at removal, with only small changes in loss and accuracy, indicating little impact on task performance.
Concern #3: It is necessary to prove that its EG residual cost and continuous deletion process bring effects, and cannot mainly rely on the broad positioning of joint optimization of training and structure.
Response: We added experiments on LLaMA-3.2-1B / IMDB using L1 and L2 norms to constrain residuals and assess the contribution of the EG rate formulation. EG achieved both better task performance and higher pruning rates than L1 and L2. We also added two ablations: gate-only learning and one-shot pruning after GRCT training. Gate-only learning produced no layers meeting the removal criterion, while one-shot pruning caused substantial performance degradation. These results support the practical value of the EG rate constraint and progressive pruning during training.
Concern #4: The text is not clear about whether these training are conducted before or after pruning, and whether the baseline starts from the same fine-tuning weights. It is recommended to provide additional explanations.
Response: All baselines undergo standard task fine-tuning with the same number of epochs: 10 for classification and 5 for generation. Each pruning baseline then applies its method starting from the same fine-tuned checkpoint. ShortGPT, LaCo, and SliceGPT are evaluated directly after pruning without recovery training; LLM-Pruner performs additional recovery fine-tuning after pruning; and SkipGPT performs router learning followed by recovery fine-tuning. We will add more detailed technical descriptions to clarify this procedure in the revised manuscript. Thank you for your valuable suggestion, which truly makes our manuscript clearer.