acceptodds
Under review as a conference paper at ICLR 2027

Learning the Optimized Transformer Structure from Generalized Residual Constrained Training

Abstract

Large language models (LLMs) are conventionally trained with a fixed, dense architecture, forcing the entire model to be optimized throughout training despite substantial structural redundancy. Post-training pruning streamlines the model structure for deployment efficiency, but it does not mitigate the training-time costs. In this paper, we propose Generalized Residual Constrained Training (GRCT), a rate–distortion training paradigm built on residual modeling that jointly optimizes task performance and model structure. By optimizing the residual constraints, GRCT directly learns an adaptive model structure during training, reducing training costs as the structure contracts and ultimately yielding a deployable compact model without an extra pruning and recovery stage. In extensive experiments, GRCT outperforms all pruning baselines at the same or higher pruning rates. Compared with the dense model, GRCT even achieves 93.8% of layer pruning while improving accuracy by 1.50 percentage points. Moreover, GRCT speeds up training and inference by up to and , with memory savings of 41.1% and 66.0% on a representative task, substantially improving time and memory efficiency of both training and inference. GRCT shifts the focus of compression research from the final model to the entire training trajectory, providing a unified path toward structure learning for modern LLMs.

Primary area:
optimization
Keywords:
Structured Pruning, Rate–Distortion Optimization, Learnable Architecture, Large Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_W3DkOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Generalized Residual Constrained Training (GRCT), a rate–distortion framework that jointly optimizes task performance and Transformer structure during end-to-end training. By modeling generalized residuals with generalized Gaussian statistics and penalizing their Exp-Golomb coding rates, GRCT adaptively suppresses and removes redundant layers, sublayers, or layer segments without requiring post-training pruning and recovery. Experiments across multiple LLMs and tasks show that GRCT preserves task performance while substantially reducing training time, memory usage, inference latency, and model size.

Critical strengths

This paper learns deployable compact models directly during training, thereby reducing both training and inference costs rather than only improving post-training efficiency. The extensive experiments, trajectory-aware efficiency metrics, and analysis of residual distributions and pruning patterns provide strong empirical support for the method’s effectiveness and adaptability across model scales and task types.

Critical weaknesses
  1. The decision to delete the paper was not directly linked to the variation in validation loss. The task loss can constrain the structural contraction, but the low gating weighted encoding cost is not directly equivalent to low task importance. The paper has not fully explained why it is sufficient as a basis for permanent deletion. Is this judgment mainly based on empirical observation?

  2. The paper permanently deletes the structure based on Formula 9 and Algorithm 1. However, the model actually incorporates the hidden state equation gr. Please explain the actual gating residual amplitude difference of each layer, as well as the variation in validation loss before and after deletion, to illustrate that the low pruning score may be due to predicting low deletion damage.

  3. The changes in regular strength and pruning intensity in the paper have not answered whether EG residual regularization is superior to other methods under the same gating training budget and physical deletion strategy. It is necessary to prove that its residual cost and continuous deletion process bring effects, and cannot mainly rely on the broad positioning of joint optimization of training and structure.

  4. The main text describes ShortGPT, LaCo, and SliceGPT as pruning baselines without recovery training, but Appendix C.1 indicates that all methods have undergone the same number of task training epochs. The text is not clear about whether these training are conducted before or after pruning, and whether the baseline starts from the same fine-tuning weights. It is recommended to provide additional explanations.

  5. The table reports training and inference acceleration, but does not list the task accuracy of these accelerated models, nor does it list the total time taken for the pruning baseline from the initial weights to the final model.

  6. The generation task is evaluated only by PPL, which cannot fully reflect the generation ability of the pruned model. It is recommended to supplement SQuAD's EM/F1, as well as the accuracy pass@1 on the code benchmark, to verify whether the model can still maintain performance under this high pruning rate.

  7. In Section 5 of the paper, teacher forcing is used to explain the excessive deletion of Attention by GRCT-SL on SQuAD. However, the current basis is mainly the pruning distribution and the differences in training/testing losses. The current explainability is insufficient. It is recommended to adjust the number of Attention deletions under the same pruning budget, or to separately adjust the regular strength of Attention and FFN to observe the generation performance.

  8. Please clarify whether the pruning rate in the cross-table results is uniformly defined. There is inconsistency in Tables 1, 2, and Figure 7, and the corresponding accuracies are not exactly the same. Please indicate whether these results come from different configurations, or different random seeds, or whether the differences in pruning rates should be verified.

Questions for authors

See Weakness.

2 comments
AuthorsOpenReview ↗

Response (1/2)

We sincerely appreciate your thorough review and thoughtful suggestions, which have helped us present our contributions more clearly and provide stronger experimental support. In this paper, we propose GRCT, a constrained training method that enables models to adaptively learn their structure based on task requirements. Thank you for recognizing GRCT’s training and inference efficiency, empirical support, and adaptability across models and tasks, as well as the value of our evaluation of structural evolution and residual analysis. Guided by your constructive feedback, we have refined the methodological explanations and added supporting experiments. We hope our responses below satisfactorily address your concerns.

Concern #1: The decision to delete the paper was not directly linked to the variation in validation loss. The task loss can constrain the structural contraction, but the low gating weighted encoding cost is not directly equivalent to low task importance. The paper has not fully explained why it is sufficient as a basis for permanent deletion. Is this judgment mainly based on empirical observation?

Response: A low residual rate actually reflects a sparse, low-magnitude residual distribution and indicates a negligible contribution to the forward hidden states and the model’s final predictions, making it a proxy for pruning. Meanwhile, GRCT continuously balances residual rate against task loss, allowing each layer’s residuals to converge smoothly and stably to a well-defined level, thereby providing a guarantee for permanent pruning based on residual rate. The table below shows the residual-RMS trajectories of retained layer L0 and pruned layer L12, with L12’s residual RMS converging to an extremely low level. We will include the trajectories of all layers in the revised manuscript.

LayerResidual 1Residual 2Residual 3Residual 4Residual 5Residual 6
L00.06060.06030.06100.04270.04260.0347
L120.12390.10960.08830.02000.01150.0056

Concern #2: Please explain the actual gating residual amplitude difference of each layer, as well as the variation in validation loss before and after deletion, to illustrate that the low pruning score may be due to predicting low deletion damage.

Response: Using training on LLaMA-3.2-1B / IMDB as an example, we report the residual RMS of each layer at its removal and the changes in loss and accuracy across the entire test set before and after removal. The results show that the residual magnitudes are already low at removal, with only small changes in loss and accuracy, indicating little impact on task performance.

MetricPruning 1Pruning 2Pruning 3Pruning 4Pruning 5Pruning 6
Step2,0542,2562,3862,4732,6303,195
Removed Layer Index121413101115
Residual RMS0.00560.00840.01030.00550.00680.0086
ΔLoss+0.00475−0.01018−0.01106−0.02363−0.04915−0.08465
ΔAccuracy−0.072%−0.008%−0.324%+0.008%−0.128%−0.012%

Concern #3: It is necessary to prove that its EG residual cost and continuous deletion process bring effects, and cannot mainly rely on the broad positioning of joint optimization of training and structure.

Response: We added experiments on LLaMA-3.2-1B / IMDB using L1 and L2 norms to constrain residuals and assess the contribution of the EG rate formulation. EG achieved both better task performance and higher pruning rates than L1 and L2. We also added two ablations: gate-only learning and one-shot pruning after GRCT training. Gate-only learning produced no layers meeting the removal criterion, while one-shot pruning caused substantial performance degradation. These results support the practical value of the EG rate constraint and progressive pruning during training.

MethodIMDB RateIMDB Accuracy
L144.3%93.22%
L239.4%93.45%
Gate-only0.0%96.03%
Post-training Pruning49.2%45.76%
GRCT49.2%93.90%

Concern #4: The text is not clear about whether these training are conducted before or after pruning, and whether the baseline starts from the same fine-tuning weights. It is recommended to provide additional explanations.

Response: All baselines undergo standard task fine-tuning with the same number of epochs: 10 for classification and 5 for generation. Each pruning baseline then applies its method starting from the same fine-tuned checkpoint. ShortGPT, LaCo, and SliceGPT are evaluated directly after pruning without recovery training; LLM-Pruner performs additional recovery fine-tuning after pruning; and SkipGPT performs router learning followed by recovery fine-tuning. We will add more detailed technical descriptions to clarify this procedure in the revised manuscript. Thank you for your valuable suggestion, which truly makes our manuscript clearer.

AuthorsOpenReview ↗

Response (2/2)

Concern #5: The table reports training and inference acceleration, but does not list the task accuracy of these accelerated models, nor does it list the total time taken for the pruning baseline from the initial weights to the final model.

Response: We have added task accuracy and total training time for Dense, GRCT, and GRCT-SL in Table 2. Taking LLaMA-3.2-1B on IMDB as an example, all five external pruning baselines start from the same Dense checkpoint, which takes 112.68 min for training. ShortGPT, LaCo, and SliceGPT then take an additional 7.57 s, 45.34 s, and 9.80 s for pruning, bringing their end-to-end times to 112.81, 113.44, and 112.84 min, respectively. LLM-Pruner takes an additional 23.03 min for pruning and recovery fine-tuning, and SkipGPT takes an additional 67.93 min for router training and recovery fine-tuning, bringing their totals to 135.71 and 180.61 min, respectively. We will report end-to-end times for all external pruning baselines in the revised manuscript so that readers can compare training cost and task accuracy together.

ModelMethodAccuracyTraining Time (min)
LLaMA-3.2-1BDense95.38%112.68
LLaMA-3.2-1BGRCT-SL93.56%108.74
LLaMA-3.2-1BGRCT93.90%93.35
Qwen3-4BDense95.74%460.37
Qwen3-4BGRCT-SL93.79%392.39
Qwen3-4BGRCT94.56%366.22
LLaMA-3.1-8BDense95.37%704.88
LLaMA-3.1-8BGRCT-SL94.26%600.10
LLaMA-3.1-8BGRCT94.45%460.03
Qwen3-14BDense96.34%1357.30
Qwen3-14BGRCT-SL94.26%1110.07
Qwen3-14BGRCT93.78%983.87

Concern #6: It is recommended to supplement SQuAD's EM/F1, as well as the accuracy pass@1 on the code benchmark, to verify whether the model can still maintain performance under this high pruning rate.

Response: We further report CodeAlpaca pass@1/Next Token Accuracy (NTA), and SQuAD EM/F1 in the table below to evaluate GRCT more comprehensively. The results show that GRCT achieves performance comparable to Dense across all metrics.

ModelMethodCodeAlpaca (pass@1)CodeAlpaca (NTA)SQuAD (EM)SQuAD (F1)
LLaMA-3.2-1BDense39.60%78.18%48.52%60.83%
LLaMA-3.2-1BGRCT35.00%75.64%52.16%64.20%
Qwen3-4BDense54.40%81.33%67.64%79.21%
Qwen3-4BGRCT48.00%77.99%69.61%80.73%
LLaMA-3.1-8BDense41.00%77.97%58.79%70.39%
LLaMA-3.1-8BGRCT39.70%77.27%59.23%71.99%
Qwen3-14BDense32.00%77.12%67.12%80.33%
Qwen3-14BGRCT35.30%78.19%63.22%75.62%

Concern #7: It is recommended to adjust the number of Attention deletions under the same pruning budget, or to separately adjust the regular strength of Attention and FFN to observe the generation performance.

Response: Thank you for your such insightful suggestion. In LLaMA-3.2-1B / SQuAD experiments, we reduced the Attention constraint strength from 0.2 to 0.01, achieving a more balanced pruning allocation between Attention and FFN. Despite its higher pruning rate, GRCT-SL with the reduced Attention constraint achieved substantially lower PPL, outperforming Dense. These results indicate that adjusting the pruning allocation between Attention and FFN can substantially improve GRCT-SL’s performance.

MethodAttention lambdaFFN lambdaPruning RatePPLAttention PrunedFFN Pruned
Dense000.0%52.770 / 160 / 16
GRCT-SL0.20.225.6%116.4311 / 164 / 16
GRCT-SL0.010.229.5%44.156 / 166 / 16

Concern #8: Please clarify whether the pruning rate in the cross-table results is uniformly defined. There is inconsistency in Tables 1, 2, and Figure 7, and the corresponding accuracies are not exactly the same. Please indicate whether these results come from different configurations, or different random seeds, or whether the differences in pruning rates should be verified.

Response: Only for Qwen3-14B / GRCT in Table 2, we slightly increased the constraint strength to λ=0.03 to narrow the pruning-rate gap between GRCT and GRCT-SL and improve their comparability of system efficiency. Apart from this adjustment, Tables 1 and 2 and Figure 7 use the same base training settings. Differences in the reported metrics arise from independent training runs, which converge to comparable levels. All parameter-pruning rates use the same metric, defined as the number of removed parameters divided by the total number of parameters before pruning, including embeddings and the LM head.

Reviewer_dfmyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Generalized Residual Constrained Training (GRCT), which jointly optimizes task performance and Transformer structure during fine-tuning. Learnable gates scale generalized residual units, and the task loss is augmented with a gate-weighted Exp-Golomb (EG) code-length surrogate. Units are removed from the computation graph when the corresponding score falls below a threshold. The framework covers sublayers, whole layers, and multi-layer segments. Experiments span four models and six tasks. The paper reports 93.8% layer pruning and 81.5% parameter pruning with a 1.50-point accuracy improvement in one Devign setting, and training and inference speedups of up to 1.53× and 4.06×.

Critical strengths

1.S1. Joint attention to structure learning and fine-tuning cost. Removing units during training physically contracts the graph and can reduce the cost of subsequent steps. Table 2 provides system evidence through lower average step time and mean per-step peak memory. 2. S2. A unified framework across pruning granularities. The same gating and rate formulation applies to Attention/FFN sublayers, whole layers, and contiguous multi-layer segments. The structural analyses in Figures 4 and 7 reveal task- and granularity-dependent retention patterns. 3. S3. Informative trajectory-aware metrics. Structural exposure and average training-time and memory metrics distinguish methods with similar final structures but different contraction timing, complementing final parameter counts.

Critical weaknesses

W1. The incremental value of the EG rate surrogate is not isolated.

The GGD observations of residual distributions motivate the EG surrogate, but the experiments do not separate its contribution from gating and progressive removal. The surrogate in Eq. 6 is a logarithmic function of residual magnitude; regularization-strength sweeps, distributional evolution, and granularity comparisons do not establish how much benefit this specific choice provides over a simple regularizer. The paper needs at least one simple regularization control, such as gate-only, in one small classification setting, such as LLaMA-3.2-1B / IMDB, reporting task quality and contraction trajectories at a comparable training budget and similar final compression level. This ablation directly tests the incremental value of the rate formulation and determines whether the surrogate should be presented as a core technical contribution or as an implementation choice.

W2. The system-efficiency results lack task accuracy for the corresponding configurations, and the baseline protocol is not fully specified.

The IMDB results in Tables 1 and 2 use different parameter-pruning rates, including 65.2% versus 69.7% for LLaMA-3.1-8B / GRCT and 55.9% versus 63.8% for Qwen3-14B / GRCT. The two experiments may use different operating points, but Table 2 does not report its associated accuracy, so readers cannot determine the quality level accompanying the reported speedups. The paper must identify the configurations used for the system measurements and provide their task accuracy.

Appendix C.1 specifies the base training epochs for all methods and the additional recovery under the official settings for LLM-Pruner and SkipGPT. In Table 1, ShortGPT and LaCo both score 37.80 on 1B / BoolQ and score 53.75 and 56.34 on IMDB, substantially below Dense. These results require a per-method description of the ordering and configurations for task fine-tuning, pruning, and recovery. It must also be confirmed that all methods use the same full-model parameter denominator for Rate, including the treatment of embeddings and the LM head. Figure 7 already uses a full-model denominator.

W3. The aggregate metric's scope must be explicit in the headline claim.

The Table 1 caption discloses that the 0.40-percentage-point average classification drop takes the better result of GRCT and GRCT-SL in each setting. Recomputing from the 16 classification settings gives average drops of approximately 1.59 and 0.83 percentage points for fixed GRCT and GRCT-SL, respectively; the variants also use different pruning rates. The 0.40 figure is therefore a per-setting upper-bound aggregate over two variants, not the performance of either fixed variant. The paper should report both variant-specific means and retain the per-setting selection qualifier in headline claims. If task-dependent variant selection is proposed, the validation-set selection rule must be specified.

Questions for authors

NA

2 comments
AuthorsOpenReview ↗

Response (1/1)

We sincerely thank you for your careful review. Your constructive comments helped us clarify our contributions and strengthen the experimental evidence. GRCT is a constrained training method that enables models to autonomously learn their structure during training according to task requirements. We appreciate your recognition of its structure learning capabilities, unified framework across pruning granularities, and evaluation of structural evolution. We carefully considered your constructive comments and supplemented the experimental evidence. We hope the responses below address your concerns.

Concern #1: The incremental value of the EG rate surrogate is not isolated. The experiments do not separate its contribution from gating and progressive removal. The paper needs at least one simple regularization control, such as gate-only, in one small classification setting, such as LLaMA-3.2-1B / IMDB.

Response: We sincerely thank you for your helpful suggestion to isolate the contribution of the EG rate. We conducted experiments on LLaMA-3.2-1B / IMDB using L1 and L2 norms to constrain residuals and assess the contribution of the EG rate formulation. The results show that EG achieved both better task performance and a higher pruning rate than L1 and L2. We also added gate-only learning and one-shot pruning after GRCT training as ablations. Gate-only learning produced no layers eligible for removal, while post-training pruning caused substantial task performance degradation. These results support the practical value of the EG rate constraint and progressive pruning during training.

MethodIMDB RateIMDB Accuracy
L144.3%93.22%
L239.4%93.45%
Gate-only0.0%96.03%
Post-training Pruning49.2%45.76%
GRCT49.2%93.90%

Concern #2: The system-efficiency results lack task accuracy for the corresponding configurations, and the baseline protocol is not fully specified. Tables 1 and 2 use different parameter-pruning rates, but Table 2 does not report the accuracy. ShortGPT and LaCo perform substantially below Dense on BoolQ and IMDB, requiring a per-method description of the ordering and configurations for task fine-tuning, pruning, and recovery. All methods use the same denominator, including embeddings and the LM head.

Response: Thank you for your valuable suggestions on task accuracy and baseline configurations. Only for Qwen3-14B / GRCT in Table 2, we slightly increased the constraint strength to λ=0.03 to narrow the pruning-rate gap between GRCT and GRCT-SL and improve the comparability of their system efficiency. Apart from this adjustment, Tables 1 and 2 and Figure 7 use the same base training settings. The other differences in IMDB results arise from independent training runs, which converge to comparable levels. All pruning rates are calculated as the number of removed parameters divided by the total number of parameters before pruning, including embeddings and the LM head. The task accuracy corresponding to each configuration in Table 2 is provided below. We will add these results to Table 2 and specify the experimental configurations in the revised manuscript.

All baselines first train with the same budget: 10 epochs for classification and 5 epochs for generation. ShortGPT, LaCo, SliceGPT, and LLM-Pruner each use 128 calibration samples for BI-based layer removal, parameter-difference merging from higher to lower layers, PCA-based hidden-dimension pruning, and Taylor importance estimation, respectively. The first three are evaluated directly after pruning without recovery training. LLM-Pruner trains for 2 epochs for recovery with rank-8 LoRA. SkipGPT trains the routers for 3 epochs, followed by 3 epochs of recovery training using rank-16 LoRA with the routers fixed. We will also include these configurations in the revised manuscript.

ModelMethodAccuracy
LLaMA-3.2-1BDense95.38%
LLaMA-3.2-1BGRCT-SL93.56%
LLaMA-3.2-1BGRCT93.90%
Qwen3-4BDense95.74%
Qwen3-4BGRCT-SL93.79%
Qwen3-4BGRCT94.56%
LLaMA-3.1-8BDense95.37%
LLaMA-3.1-8BGRCT-SL94.26%
LLaMA-3.1-8BGRCT94.45%
Qwen3-14BDense96.34%
Qwen3-14BGRCT-SL94.26%
Qwen3-14BGRCT93.78%

Concern #3: The paper should report both variant-specific means and retain the per-setting selection qualifier in headline claims.

Response: Thank you for your valuable suggestions on reporting the aggregate metrics, which help us present our experimental findings more accurately and clearly. The caption of Table 1 specifies that the average accuracy drop of 0.40 percentage points is calculated by taking the better result between GRCT and GRCT-SL in each setting. We will further report in the main text and experimental section that GRCT and GRCT-SL achieve average accuracy drops of 1.59 and 0.83 percentage points, with average pruning rates of 48.6% and 47.6%, respectively, to present the results and conclusions more clearly and comprehensively.

Dear Reviewer dfmy,

Thank you again for your careful review and constructive feedback, and for recognizing GRCT’s structure learning capabilities, unified framework across pruning granularities, and evaluation of structural evolution. Your suggestions have helped us strengthen the experimental evidence and clarify the presentation of our results.

In our response, we added L1, L2, gate-only, and post-training pruning ablations to examine the contribution of the EG rate formulation and progressive pruning. We also provided the task accuracy corresponding to the system-efficiency configurations, clarified the per-method training, pruning, and recovery protocols, and confirmed that all pruning rates use the same full-model parameter denominator. Finally, we committed to reporting the variant-specific average accuracy drops and pruning rates more explicitly, while retaining the per-setting selection qualifier for the 0.40-percentage-point aggregate. We hope these clarifications and additional results have addressed your concerns. If you have any remaining questions or concerns, we would be grateful if you could let us know, so that we can provide further clarification or additional experiments before the discussion period ends.

If our responses have satisfactorily addressed your concerns, we would sincerely appreciate it if you could consider improving your score in light of these clarifications and new results. Thank you again for your time and valuable feedback.

Best regards, The Authors

Reviewer_TFLHOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper finds that Transformer residual updates follow a generalized Gaussian distribution, so each layer’s redundancy can be measured by its residual coding rate. It proposes GRCT, which jointly optimizes task loss and this rate so the model prunes itself during training and directly yields a compact model. GRCT removes up to 93.8% of layers, improves accuracy by 1.50 points over the dense model, and speeds up inference by up to 4.06×.

Critical strengths
  1. The idea sounds feasible and there are no problems with it.
  2. The writing is clear and concise.
Critical weaknesses
  1. Training-time pruning is not a new research direction. Structural regularization / Learnable gating / Accelerating training via pruning are not new methods. In the small-model era, these ideas were already studied extensively, and the authors do not introduce a new insight here. Merely applying an existing idea to LLMs is not sufficient for acceptance at a top-tier conference in my view.

  2. The maximum sequence length in the experiments is only 512 tokens. It is unclear whether the method remains effective in long-context / agentic settings, which makes its practical usefulness hard to accept.

  3. Qwen3 / LLaMA-3.2 already contain substantial redundancy. It is also unclear whether the method would remain effective on hybrid models (e.g., Qwen3.5)

Reference:

  1. Learning Structured Sparsity in Deep Neural Networks, NIPS 2016
  2. Learning Efficient Convolutional Networks through Network Slimming, ICCV2017
  3. PruneTrain: Fast Neural Network Training by Dynamic Sparse Model Reconfiguration, SC 2019
  4. Unified Visual Transformer Compression, ICLR 2022
  5. Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning, ICLR 2024
Questions for authors

Please see weakness

3 comments
AuthorsOpenReview ↗

Response (1/2)

Thank you for reviewing our manuscript and for your thoughtful feedback. We propose GRCT, a method that enables a model to learn a task-adapted architecture during training without a predefined target. Through generalized Gaussian residual modeling and continuous residual compression as structured pruning, GRCT balances task performance and structural compactness in a single rate–distortion objective, enabling the model to jointly optimize its parameters and architecture through end-to-end training and yield an effective, compact model. We appreciate your recognition of GRCT’s feasibility and your remarks on the manuscript’s clarity and conciseness. We considered your concerns and conducted additional experiments in response. We hope the clarifications and new results below are helpful.

Concern #1: Training-time pruning is not a new research direction. Structural regularization / Learnable gating / Accelerating training via pruning are not new methods. In the small-model era, these ideas were already studied extensively, and the authors do not introduce a new insight here. Merely applying an existing idea to LLMs is not sufficient for acceptance at a top-tier conference in my view.

Response: We reviewed the references you provided [1–5], which contribute to structured pruning through structural regularization [1,3], gating [2,4], and training acceleration [5], targeting layers [1,4,5] or channels [2,3]. We also find these studies differ substantially from our GRCT in focus and contributions. We would like to elaborate on these distinctions to clarify GRCT’s contribution and address your concerns more fully:

  1. GRCT is structure-constrained training, rather than training-time pruning. Training-time pruning [5] typically relies on predefined target architecture and layer contribution scores to alternate between pruning and recovery training. In contrast, GRCT defines a unified rate–distortion objective that enables the model to adaptively optimize its architecture through end-to-end training. Under this trade-off, the residual contributions of certain layers progressively diminish until those layers are naturally removed, yielding a model whose size matches task requirements without a predefined pruning target.
  2. GRCT trains models through rate–distortion optimization, whose mechanism differs from structural regularization. Structural regularization [1,3] typically penalize weight norms to drive all parameters in a layer toward zero for pruning; the regularization term itself does not explicitly account for task requirements. In contrast, GRCT regulates each layer’s contribution by constraining the coding rate of residuals generated from task inputs, without requiring all layer weights to become zero. By balancing residual coding rate against cross-entropy loss, GRCT adaptively determines how much to compress each layer’s residual contribution according to task requirements, enabling task-aware structural optimization. Furthermore, layer-wise structural regularization may substantially degrade LLM performance. As shown in the table below, applying methods [1,3] during LLM training leads to considerable performance degradation across all evaluated datasets, suggesting that their success on ResNet does not directly transfer to LLMs in our experimental setting.
  3. The original contributions of GRCT are Transformer structure learning, compression-as-pruning paradigm, and generalized residual modeling. In GRCT, learnable gating serves as an auxiliary component for constructing the learnable architecture and its objective function, while training and inference acceleration are reported as experimental benefits to assess the efficiency gains from structural optimization. They are not claimed as an independent contribution in this work.

Furthermore, we extended methods [2,3] to layer pruning and compared methods [1–5] with GRCT on LLaMA-3.1-8B across six datasets. As shown below, GRCT’s adaptive structure learning achieves a better trade-off between task performance and model structure.

MethodBoolQ Rate / AccuracyPIQA Rate / AccuracyDevign Rate / AccuracyIMDB Rate / AccuracyCodeAlpaca Rate / PPLSQuAD Rate / PPL
SSL [1]49.4% / 74.74%58.1% / 62.24%81.4% / 65.41%63.9% / 85.37%32.6% / 3.6527.2% / 207.26
Network Slimming [2]49.4% / 79.76%58.1% / 69.75%81.4% / 63.95%63.9% / 92.54%32.6% / 3.7627.2% / 152.25
PruneTrain [3]49.4% / 64.98%58.1% / 50.38%81.4% / 64.35%63.9% / 83.82%32.6% / 3.7227.2% / 221.20
UVC [4]49.4% / 81.62%58.1% / 68.99%81.4% / 64.90%63.9% / 91.64%32.6% / 3.8927.2% / 162.05
Sheared LLaMA [5]49.4% / 77.89%58.1% / 70.89%81.4% / 65.48%63.9% / 92.16%32.6% / 3.4927.2% / 170.76
GRCT51.6% / 85.35%59.8% / 75.14%81.5% / 65.67%65.2% / 94.10%32.6% / 2.6227.2% / 51.10
AuthorsOpenReview ↗

Response (2/2)

Concern #2: The maximum sequence length in the experiments is only 512 tokens. It is unclear whether the method remains effective in long-context / agentic settings, which makes its practical usefulness hard to accept.

Response: Thank you for your attention to the sequence length setting. The 512-token limit actually applies only to classification tasks and generation sequences are not truncated. We adopted this setting because it has little impact on classification performance while helping standardize evaluation conditions and improve comparability across tasks and models. Additional LLaMA-3.2-1B experiments without this limit during classification training show little change in results. We will clarify this setting and include these results in the manuscript.

MethodMetricBoolQPIQADevignIMDB
GRCT (Full Length)Pruning Rate44.3%44.3%14.8%49.2%
GRCT (Full Length)Accuracy78.72%74.54%67.48%94.80%
GRCT (512 Token)Pruning Rate39.4%44.3%24.6%49.2%
GRCT (512 Token)Accuracy79.85%72.74%67.20%93.90%

Concern #3: Qwen3 / LLaMA-3.2 already contain substantial redundancy. It is also unclear whether the method would remain effective on hybrid models (e.g., Qwen3.5).

Response: Thank you for your suggestion to evaluate GRCT on hybrid architectures. GRCT adaptively optimizes model architecture according to task requirements, balancing task performance and structural complexity without being limited to a specific architecture. Additional experiments on the hybrid model Qwen3.5 show that GRCT also effectively prunes this hybrid architecture.

ModelIMDB RateIMDB Accuracy
Dense0%93.104%
GRCT81.25%93.300%

Dear Reviewer TFLH,

Thank you again for carefully reviewing our paper and for recognizing the feasibility of our approach and the clarity and conciseness of our writing. We greatly appreciate your feedback and hope that our responses have helped address your concerns.

In our responses, we further clarified GRCT’s core contributions and research positioning, as well as its distinctions from existing approaches such as structured pruning and structural regularization. We also provided comparative experiments with the methods in the references you suggested to demonstrate GRCT’s effectiveness. In addition, we included experiments without the 512-token classification length limit and results on the hybrid Qwen3.5 architecture. We hope these explanations and additional results have addressed your concerns. If you have any remaining questions or concerns, we would be grateful if you could let us know, so that we can provide further clarification or additional experiments before the discussion period ends.

If our responses have satisfactorily addressed your concerns, we would sincerely appreciate your reconsideration of the assessment and score in light of these clarifications and new results. Thank you again for your time and valuable feedback.

Best regards, The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Contributions and Rebuttal Updates

Dear Area Chair and Reviewers,

We sincerely thank you for the time and effort you have devoted to evaluating our submission and for organizing and coordinating the review process. We also greatly appreciate the reviewers’ careful evaluations and constructive comments, which have helped us clarify the core contributions of our method and strengthen the supporting evidence through additional experiments. In our rebuttal, we addressed the reviewers’ main concerns individually and provided more detailed methodological explanations and experimental results. To facilitate your overall assessment, we briefly summarize GRCT’s core innovation and the clarifications and additional experimental findings provided in response to each reviewer.

GRCT’s innovation and contributions. In this paper, we propose GRCT, which enables Transformers to jointly optimize model parameters and task-adaptive structures during training through generalized residual modeling and a unified rate–distortion objective. Unlike training-time pruning approaches that rely on a predefined target architecture, GRCT constrains structural complexity through residual coding rates, which, together with the task loss, determine the extent to which each structural unit is retained. Its “compression-as-pruning” mechanism progressively diminishes the residual contributions of redundant units and directly removes these units during training, contracting the computation graph. Consequently, GRCT not only directly yields a deployable compact model but also reduces subsequent training costs as the structure contracts, without requiring a separate pruning and recovery stage.

Clarifications and additional experiments in response to each reviewer.

  • For Reviewer TFLH: We clarified the distinctions between GRCT and structural regularization, learnable gating, and conventional training-time pruning, and added comparisons with all five reviewer-suggested methods on LLaMA-3.1-8B across six datasets. GRCT achieved better task performance at comparable or higher pruning rates in every setting, supporting the effectiveness of the proposed method. Furthermore, classification experiments without sequence truncation yielded similar results, while experiments on the hybrid Qwen3.5 architecture achieved 81.25% parameter pruning without an accuracy drop, further demonstrating GRCT’s applicability.

  • For Reviewer dfmy: We added ablations using L1, L2, gate-only learning, and one-shot pruning after training. The EG constraint achieved both better accuracy and higher pruning rates; gate-only learning produced no removable units, while deferring pruning until after training substantially degraded performance. These findings support the practical value of the EG residual constraint and progressive removal during training. We also provided the accuracy associated with the system-efficiency configurations, clarified the baseline training, pruning, and recovery protocols and the uniform definition of parameter-pruning rate, and explicitly distinguished the variant-specific means from the aggregate obtained by selecting the better variant in each setting.

  • For Reviewer W3Dk: We added residual-amplitude trajectories and measurements of task loss and accuracy immediately before and after structural removal. Residual amplitudes were already low at removal, and the immediate accuracy changes were small, providing direct empirical evidence supporting the removal of units with low residual contributions. We also supplemented accuracy, total training time, and end-to-end costs for the external baselines, allowing efficiency gains to be assessed alongside task quality, and added pass@1/NTA and EM/F1 metrics to broaden the evaluation of generation capabilities. Further experiments adjusting the Attention/FFN constraint allocation showed that more balanced structural contraction can substantially improve generation-task performance at a higher pruning rate.

We believe these clarifications and additional experiments further strengthen GRCT’s novelty and empirical support. We will incorporate the corresponding results and experimental details into the revised manuscript, and respectfully ask that you consider this additional evidence in your overall assessment. Once again, we sincerely thank you and the reviewers for the time and effort devoted to our submission and for the valuable suggestions that have helped improve this work.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.