acceptodds
Under review as a conference paper at ICLR 2027

Correlation-Aware Structured Pruning for Large Language Models

Abstract

Structured pruning is a promising approach for reducing the substantial inference costs of Large Language Models (LLMs) while maintaining hardware efficiency. Existing methods often rely on independent unit scores or capture cross-unit dependencies only implicitly, which may misestimate the joint effect of pruning correlated units. To address this, we propose a Correlation-Aware Structured Pruning method. We explicitly decompose the set-level reconstruction error into individual pruning costs and pairwise interactions that capture dependencies in activations and weights, leading to a cardinality-constrained binary quadratic program. Since this binary quadratic program is NP-hard and difficult to solve exactly, we develop a greedy interaction algorithm based on dependency-aware marginal costs to optimize unit selection. Furthermore, we incorporate a gradient-based strategy to achieve adaptive layer-wise sparsity allocation across the entire model. Extensive experiments on mainstream LLMs demonstrate that incorporating correlation information yields competitive accuracy-efficiency trade-offs compared to representative structured pruning baselines.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Inference Acceleration, Structured Pruning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_dHWeOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Correlation-Aware Structured Pruning (CASP), a post-training method that explicitly models dependencies between prunable units in large language models. It formulates structured pruning as a cardinality-constrained binary quadratic program by decomposing the reconstruction error into individual costs and pairwise interaction terms, then solves it using an efficient greedy interaction algorithm with gradient-based layer-wise sparsity allocation. Experiments on multiple mainstream LLMs demonstrate that CASP achieves competitive accuracy–efficiency trade-offs, particularly under high pruning ratios.

Critical strengths

This paper's greedy interaction algorithm is computationally efficient, naturally supports both FFN channel pruning and attention-head pruning, and is complemented by an adaptive layer-wise sparsity allocation strategy. The experimental evaluation is comprehensive, covering multiple models, pruning ratios, benchmarks, and ablation studies that validate the contribution of each component.

Critical weaknesses
  1. The method proposed in this paper still falls within the main line of post-training structured pruning, and the only difference from OSSCAR/SlimGPT/LLM-Pruner is that it explicitly formalizes the implicit dependencies and solves them better. Therefore, the innovation of the method in this paper is a major issue.

  2. In the paper, OSSCAR jointly optimizes the structure set and the retained weights; CASP is post-training pruning, constructing Q and only selecting the masks without updating the retained weights. However, the distinction between implicit and explicit in the paper is not sufficient. When comparing the same reconstruction targets, the incremental effect relative to the existing related perception methods is not sufficiently separated. Therefore, the actual method effect needs to be further clarified.

  3. In the GQA experiment of the paper, the pruning rules, pruning efficiency and model performance are reported, but the actual structure after pruning and the hardware efficiency after pruning are not given. There is no report of actual parameter quantity, the number of qkv per layer, and the comparison should be made fairly with other methods under the same actual parameter quantity or computing budget.

  4. Table 3 shows that the budget allocation between layers will affect the final effect. It should be fixed at the same layer-wise budget to compare CASP, SlimGPT, and OSSCAR again, to distinguish the contributions of the interaction selection and allocation strategies.

  5. In Appendix Figure 2, the curves are presented as continuous relaxation lower bounds, but the specific calculation method of the reconstruction error and the normalization method are not explained. Whether the same Q is used for each pruning rate.

  6. In the experiments in Table 2 when pruning the attention heads, it is unclear how they are operated specifically: If the channel-level Q is diagonalized first, the interaction within the same head will also be deleted.

Questions for authors

See Weakness.

3 comments
AuthorsOpenReview ↗

Rebuttal for Reviewer dHWe (1/2)

Dear Reviewer dHWe,

Thank you for identifying where the current comparisons mix allocation, mask selection, and weight recovery. We address these separately below.

1. Scope of the contribution.

Thank you for raising this important question, which goes to the core of our contribution. We respectfully clarify that the premise of a single difference does not hold: CASP is not a point-wise reformulation but a tightly coupled pipeline of three components.

  • Quadratic reconstruction and greedy selection. We use LSA's quadratic reconstruction criterion to separate individual removal costs from pairwise interactions and apply its incremental greedy rule.
  • Interaction-aware structured pruning. We tie the mask decisions for dimensions that must be removed together and aggregate their pairwise interactions into an exact reconstruction objective for complete units such as attention heads.
  • Global gradient-based sparsity allocation. We allocate sparsity across layers using gradients of the end-to-end loss, so each local reconstruction problem has a budget informed by the layer's effect on model-level performance.

At 50% pruning, CASP reaches 49.31% average accuracy, compared with 44.33% for Diag in the selection ablation and 45.17% for the strongest alternative schedule in the allocation ablation (Tables 2 and 3 of the submitted paper). With matched per-layer budgets, CASP also achieves a PTB perplexity of 56.61, compared with 63.13 for SlimGPT and 76.06 for OSSCAR (Table 3.3). Together, these results show the value of combining interaction-aware selection with allocation based on end-to-end loss, particularly at higher sparsity.

2. Weight compensation.

OSSCAR jointly optimizes the mask and retained weights, and CASP selects the mask by explicitly modeling pairwise interactions in the fixed-weight reconstruction objective. Refitting weights can lower local reconstruction error, with a risk of overfitting on a small calibration set.

Table 3.1. CASP and OSSCAR under matched per-layer budgets on LLaMA2-7B.

TargetMethodWikiText2 PPL ↓PTB PPL ↓Avg. accuracy (%) ↑
20%OSSCAR6.9735.6363.04
20%CASP6.7024.5263.27
50%OSSCAR18.4176.0647.82
50%CASP15.2656.6149.31

Table 3.1 shows that under matched per-layer budgets, CASP outperforms OSSCAR on all three metrics at both pruning ratios, lowering PTB perplexity by 31.2% at 20% pruning and 25.6% at 50%.

In addition, as reported in Appendix B.6, LoRA fine-tuning on additional training data raises CASP's average accuracy from 49.31% to 53.43% at 50% pruning. This approach may offer more robust recovery than weight compensation on a small calibration set.

3. GQA structure and hardware efficiency.

Table 3.2 reports parameter counts and inference measurements for physically pruned LLaMA3-8B-Base models at comparable parameter counts. CASP prunes query heads and removes a shared key/value head only when all its associated query heads have been pruned. The target column denotes CASP’s nominal pruning ratio. We report MACs for batch size 1 and sequence length 128.

Table 3.2. Retained attention heads and inference efficiency on LLaMA3-8B-Base.

TargetMethodRetained Q/K/V headsParams (B)MACs (G)Prefill (s) ↓Decode (s) ↓Peak GPU (GiB) ↓
0%Dense1024 / 256 / 2568.0303964.890.12441.972016.50
20%LLM-Pruner816 / 204 / 2046.6790791.060.10491.800313.91
20%LoRAP (GQA)1024 / 256 / 2566.6786791.880.10961.966513.99
20%CASP819 / 246 / 2466.6785791.010.10951.938814.00
50%LLM-Pruner608 / 152 / 1524.6019524.320.07711.69079.95
50%LoRAP (GQA)1024 / 256 / 2564.6040526.330.08511.978310.24
50%CASP512 / 186 / 1864.6028524.030.07791.766810.02

Head counts are summed across layers. LoRAP uses low-rank compression and preserves the original head counts.

At both pruning targets, the methods have comparable parameter counts and MACs (Table 3.2). CASP's prefill and decode latencies and peak GPU memory are close to LLM-Pruner's. Compared with LoRAP, CASP records lower prefill and decode latencies at both targets, with nearly identical peak GPU memory at 20% and lower peak memory at 50%. At the 50% target, CASP achieves the lowest perplexity on WikiText2 and PTB (33.25 and 53.03) and the highest average zero-shot accuracy (45.20%, versus 40.60% for LLM-Pruner and 43.93% for LoRAP), as reported in Table 10 of the submitted paper. These results show that CASP retains higher model quality while keeping inference costs competitive. We will report the full GQA hardware comparison in a new appendix section once the remaining pruning ratios are evaluated.

AuthorsOpenReview ↗

Rebuttal for Reviewer dHWe (2/2)

4. Shared per-layer budgets.

On LLaMA2-7B, SlimGPT and OSSCAR use CASP's per-layer attention-head and FFN-channel budgets while retaining their own selection and weight compensation procedures. In Table 3.3, the rows marked “Original” show the baselines under their original budgets. These results and the CASP results are from Table 1 of the submitted paper. Accuracy is averaged over seven tasks.

Table 3.3. Pruning results under matched per-layer budgets on LLaMA2-7B.

TargetMethodWikiText2 PPL ↓PTB PPL ↓Avg. accuracy (%) ↑
20%SlimGPT (Original)7.2929.6863.01
20%SlimGPT6.8433.3063.24
20%OSSCAR (Original)7.0326.6763.08
20%OSSCAR6.9735.6363.04
20%CASP6.7024.5263.27
50%SlimGPT (Original)18.3662.3648.04
50%SlimGPT16.6163.1349.04
50%OSSCAR (Original)20.4360.9735.48
50%OSSCAR18.4176.0647.82
50%CASP15.2656.6149.31

First, at 50% pruning, CASP's sparsity allocation improves average accuracy for both SlimGPT and OSSCAR, from 48.04% to 49.04% and from 35.48% to 47.82%, respectively (Table 3.3). Second, under the same per-layer budgets, CASP's interaction-aware pruning achieves the lowest PTB perplexity at 20% pruning and the best results on all three metrics at 50%. We will report the full matched-budget comparison in a new appendix section once the remaining pruning ratios are evaluated.

5. Figure 2 calculation and normalization.

All sparsities use the same QQ from the first FFN layer of LLaMA2-7B. Each mask is scored by z⊤Qzz^\top Qz, the mean squared Frobenius reconstruction error over calibration samples, without further normalization. The lower bound minimizes the same objective over the continuous relaxation. We have clarified how the error is calculated in Appendix A.1 (p. 13) and specified the error scale in the Figure 2 caption (p. 14).

6. Attention-head Diag ablation.

Diag takes the diagonal after aggregating channel interactions by head (Section 4.2). This preserves within-head interactions and removes only between-head interactions. We have clarified this in the Interaction-Aware Selection paragraph of Section 5.3 (p. 9).

AuthorsOpenReview ↗

Follow-up on Our Rebuttal to Reviewer dHWe

Dear Reviewer dHWe,

Thank you for your valuable feedback on CASP’s methodological contribution and experimental evaluation. We have clarified the method’s contribution and relationship to prior work, and added experiments under matched per-layer budgets and on GQA hardware efficiency. We believe these changes clarify our contribution and strengthen the experimental evidence. With the discussion period ending on November 18, 2026 (AoE), we would greatly appreciate your thoughts on whether our responses address your concerns.

Reviewer_FHKVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Correlation-Aware Structured Pruning (CASP) for large language models to explicitly model cross-unit dependencies by decomposing the reconstruction error into individual pruning costs and pairwise interactions. To efficiently optimize pruning decisions, the authors develop a greedy interaction algorithm and a gradient-based strategy for adaptive layer-wise sparsity allocation. Extensive experiments demonstrate that CASP achieves competitive performance compared with existing structured pruning methods, particularly at high pruning ratios.

Critical strengths

1.The paper explicitly models cross-unit dependencies by decomposing the reconstruction error into individual pruning costs and pairwise interactions, providing a clear mathematical foundation for correlation-aware structured pruning. 2.The proposed greedy interaction algorithm efficiently incorporates pairwise dependencies into pruning decisions, reducing the search complexity to O(km) once the interaction matrix is constructed.

Critical weaknesses

1.The proposed method relies on a layer-wise greedy search and the construction of dense interaction matrices, potentially introducing substantial computational and memory overhead. Although the paper reports pruning runtime, it lacks a direct comparison of computational efficiency with existing structured pruning baselines. Such comparisons are necessary to assess the practical efficiency and scalability of the proposed method. 2.The paper presents the proposed greedy algorithm as one of its main contributions. However, similar greedy selection strategies have already been explored in prior pruning literature [1]. The authors should clearly distinguish their approach from existing methods and further substantiate its methodological novelty. 3.The experimental evaluation omits an important baseline for structured pruning of LLMs [2]. References: [1]Kwon et al. A Fast Post-Training Pruning Framework for Transformers. NeurIPS 2022. [2] He et al. Olica: Efficient Structured Pruning of Large Language Models without Retraining. ICML 2025.

Questions for authors

Please refer to the weaknesses. If all my concerns are adequately addressed, I would recommend acceptance.

4 comments
AuthorsOpenReview ↗

Rebuttal for Reviewer FHKV (1/2)

Dear Reviewer FHKV,

Thank you for highlighting the explicit interaction model and asking how the offline cost and methodological contribution compare with prior work.

1. Pruning time and memory.

Table 2.1 reports offline pruning time, peak GPU memory, and peak CPU resident memory on LLaMA2-7B at 20% and 50% target pruning. Following their official implementations, we keep the model on the CPU for SlimLLM and LLM-Pruner during pruning. SlimLLM also uses GPU memory during its run.

Table 2.1. Offline pruning time and peak memory on LLaMA2-7B.

TargetMethodTime (s) ↓Peak GPU memory (GiB) ↓Peak CPU RSS (GiB) ↓
20%OSSCAR425.8823.272.02
20%FLAP49.6717.761.75
20%LoRAP317.4118.911.99
20%SlimGPT197.1420.901.91
20%LLM-Pruner60.250.0043.88
20%SlimLLM671.494.1322.63
20%CASP185.3329.261.75
50%OSSCAR533.6721.862.02
50%FLAP49.6717.761.75
50%LoRAP424.5218.912.03
50%SlimGPT238.0820.771.91
50%LLM-Pruner59.830.0041.94
50%SlimLLM731.464.2022.64
50%CASP186.5929.261.76

FLAP and LLM-Pruner prune faster and have lower peak GPU memory use than CASP, which takes about three minutes at both pruning ratios (Table 2.1). LLM-Pruner keeps the model on the CPU, with a peak CPU RSS of 43.88 GiB at 20% and 41.94 GiB at 50%. CASP uses GPU memory to compute gradients and construct interaction matrices during offline pruning, and releases those allocations afterward. Table 2.2 reproduces the corresponding quality results for these three methods at both pruning ratios from Table 1 of the submitted paper.

Table 2.2. Model quality on LLaMA2-7B for selected methods (from Table 1 of the submitted paper).

TargetMethodWikiText2 PPL ↓PTB PPL ↓Avg. accuracy (%) ↑
20%FLAP7.1228.0057.05
20%LLM-Pruner10.5542.7859.77
20%CASP6.7024.5263.27
50%FLAP24.95143.2541.69
50%LLM-Pruner97.96377.5041.10
50%CASP15.2656.6149.31

At 20% and 50% pruning, CASP improves average accuracy over FLAP by 6.22 and 7.62 percentage points and over LLM-Pruner by 3.50 and 8.21 points, respectively. At 50% pruning, CASP also achieves over 84% lower perplexity on both corpora than LLM-Pruner. These gains show the quality benefit of CASP's one-time offline cost. We will extend the existing pruning-time appendix section with the full time and memory comparison once the remaining pruning ratios are evaluated.

2. Relation to Kwon et al. (2022).

Thank you for pointing us to this work. Table 2.3 compares how the two methods model interactions among pruned units and select masks.

Table 2.3. Comparison of Kwon et al. (2022) and CASP.

AspectKwon et al. (2022)CASP
Modeling objectiveApproximate end-to-end loss changes with empirical Fisher.Expand local reconstruction error into individual costs and pairwise interactions.
Greedy strategyRefine an initial mask with one-for-one swaps.Build the mask incrementally by adding the lowest-cost removal at each step.
Models evaluatedBERT-base and DistilBERTDecoder-only LLMs (e.g., LLaMA2 and Qwen3)

Scaling Kwon's pipeline to LLMs may pose computational challenges, particularly for Fisher estimation and mask tuning. We have cited Kwon et al. [1] in Section 3.

Incremental greedy selection is a standard optimization heuristic [2] and is used in LSA to approximate the solution to the quadratic reconstruction problem [3]. CASP adapts this formulation to structured LLM pruning by aggregating pairwise interactions into an exact group-level criterion for units such as attention heads. This group-level criterion is stated explicitly in the revised introduction.

[1] Woosuk Kwon, Sehoon Kim, Michael W. Mahoney, Joseph Hassoun, Kurt Keutzer, and Amir Gholami. 2022. A Fast Post-Training Pruning Framework for Transformers. NeurIPS.

[2] G. L. Nemhauser, L. A. Wolsey, and M. L. Fisher. 1978. An Analysis of Approximations for Maximizing Submodular Set Functions—I. Mathematical Programming, 14, 265–294.

[3] Zhiguo Yang, Changjian Deng, Qinke Chen, Zijing Zhou, and Jian Cheng. 2026. LSA: Layer-wise Sparsity Allocation for Large Language Model Pruning Based on Minimal Linear Reconstruction Error. International Conference on Learning Representations (ICLR).

AuthorsOpenReview ↗

Rebuttal for Reviewer FHKV (2/2)

3. Olica baseline.

Thank you for suggesting Olica [4]. Table 2.4 compares Olica and CASP on LLaMA2-7B at 20% and 50% target pruning. The CASP scores are from the submitted paper, and accuracy is averaged over seven tasks.

Table 2.4. Comparison with Olica on LLaMA2-7B.

TargetMethodWikiText2 PPL ↓PTB PPL ↓Avg. accuracy (%) ↑
20%Olica6.6329.3462.54
20%CASP (paper)6.7024.5263.27
50%Olica16.76221.4446.88
50%CASP (paper)15.2656.6149.31

At 50% pruning, CASP reduces PTB perplexity by 74.4% and WikiText2 perplexity by 8.9% relative to Olica, while improving average accuracy by 2.43 percentage points (Table 2.4). Olica leads only on WikiText2 perplexity at 20% pruning, where CASP's perplexity is about 1% higher. Overall, CASP leads on five of six comparisons, with a clear advantage at 50% pruning. We will add Olica to the main LLaMA2-7B results table once the remaining pruning ratios are evaluated.

[4] Jiujun He and Huazhen Lin. 2025. Olica: Efficient Structured Pruning of Large Language Models without Retraining. ICML, PMLR 267, 22580–22594.

Thank you for the detailed responses. I appreciate the effort you put into addressing my concerns, particularly the cost comparisons. The clarification regarding Kwon et al. (2022) and the performance comparisons with Olica are convincing. I am therefore increasing my rating from 2 to 3. Please incorporate these discussions into your final version.

AuthorsOpenReview ↗

Response to Reviewer FHKV’s Follow-up Comments

Dear Reviewer FHKV,

We appreciate your constructive comments on CASP’s computational cost, its connection to Kwon et al. (2022), and Olica, as well as your recognition of our additional experiments and clarifications. We believe your feedback has strengthened the presentation of our contribution and the trade-off between pruning cost and model quality. The revised manuscript will include the completed experimental results and accompanying discussions.

We also noticed that the review form still shows a rating of 2, while your follow-up comment mentions an increase to 3. When convenient, could you please check whether the updated rating has been saved in the system? Thank you again for your time and consideration.

Reviewer_TZbgOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CASP combines gradient-based layerwise sparsity allocation with correlation-aware selection of attention heads and FFN channels. It incorporates the interactions between jointly removed units into a quadratic reconstruction objective and uses incremental greedy selection. Experiments are performed on perplexity and zero-shot accuracy with ablations of the reconstruction criterion and allocation strategy.

Critical strengths
  1. Clear formulation. The reconstruction objective clearly captures the way in which jointly removed units can either amplify or cancel out reconstruction errors, with a simple procedure for updating their marginal removal costs.
  2. Useful component ablations. Tables 2 and 3 examine off-diagonal interactions and layerwise allocation separately, respectively, confirming the empirical benefit of both components over the alternatives considered.
Critical weaknesses
  1. Unclear novelty. Reconstruction-based selection, interaction-aware scoring, and non-uniform allocation have close precedents in OSSCAR [1], LSA [2], and OWL [3]. The additional technical contribution of CASP’s formulation and combination is not sufficiently clear.
  2. Limited baseline, model, and evaluation coverage. The comparisons do not adequately cover recent non-uniform pruning methods. The evaluation also relies on older model families and a narrow range of tasks, limiting the evidence for the method’s broader effectiveness.
Questions for authors
  1. Technical contribution. Can the authors be more specific about what the reconstruction criterion and greedy procedure add to existing reconstruction-based pruning? In particular, how do they relate to the very similar formulation in LSA [2]?
  2. Non-uniform baselines. Please extend the discussion and comparisons to recent works, for example, LSA and OWL for non-uniform pruning, and GISP [4] for gradient-based non-uniform pruning.
  3. Model and task coverage. Please extend the evaluation to a recent model family such as Qwen3, and include MMLU for broader knowledge evaluation and GSM8K for generation-based reasoning.
  4. Allocation benefit and cost. How does gradient-based allocation compare with a simple schedule that preserves the first 10% of layers and the final layer while pruning uniformly elsewhere? Also, what is the cost (e.g., memory, time) of gradient-based allocation compared with other design choices?

[1] OSSCAR: One-Shot Structured Pruning in Vision and Language Models with Combinatorial Optimization. ICML 2024.

[2] LSA: Layer-wise Sparsity Allocation for Large Language Model Pruning Based on Minimal Linear Reconstruction Error. ICLR 2026.

[3] Outlier Weighed Layerwise Sparsity (OWL): A Missing Secret Sauce for Pruning LLMs to High Sparsity. ICML 2024.

[4] From Local to Global: Revisiting Structured Pruning Paradigms for Large Language Models. ACL 2026.

4 comments
AuthorsOpenReview ↗

Rebuttal for Reviewer TZbg (1/2)

Dear Reviewer TZbg,

Thank you for recognizing the value of the reconstruction formulation and component ablations. We address the scope of the contribution, the choice of baselines, and the benefit and cost of the allocation rule below.

1. Technical contribution and relation to LSA.

Thank you for pointing out the overlap with LSA. After reviewing LSA, we agree that it already presents the quadratic reconstruction objective for input-channel pruning and the incremental greedy update. We have revised the paper’s framing, contribution statements, and method description, and added citations to LSA where we discuss the objective and the greedy update.

LSA primarily uses this formulation to allocate sparsity for unstructured pruning. CASP adapts the formulation to structured pruning by aggregating pairwise interactions into an exact objective for selecting complete units such as attention heads. A further contribution is our gradient-based global sparsity allocation across Transformer blocks.

2. Non-uniform baselines.

On LLaMA2-7B, we compare LSA and OWL with the simple schedule requested in Q4 (Protected Uniform), which keeps the first 10% of layers and the final layer dense and prunes the rest uniformly.

CASP scores in Table 1.1 are from the submitted paper. Accuracy is averaged over seven tasks.

Table 1.1. Layer-wise sparsity allocation on LLaMA2-7B.

TargetAllocationWikiText2 PPL ↓PTB PPL ↓Avg. accuracy (%) ↑
20%LSA6.8526.6062.96
20%OWL6.9624.9262.59
20%Protected Uniform6.8225.3561.89
20%CASP (paper)6.7024.5263.27
50%LSA22.89118.4241.79
50%OWL20.19106.5342.64
50%Protected Uniform21.8785.1040.51
50%CASP (paper)15.2656.6149.31

GISP determines layer sparsity through iterative global pruning, so its allocation is coupled to its selector. Table 1.2 therefore compares GISP as a standalone baseline at the same target pruning ratios, using CASP scores from the submitted paper and GISP scores from our additional runs.

Table 1.2. Comparison with GISP on LLaMA2-7B.

TargetMethodWikiText2 PPL ↓PTB PPL ↓Avg. accuracy (%) ↑
20%GISP7.2125.1358.32
20%CASP (paper)6.7024.5263.27
50%GISP25.4196.9441.85
50%CASP (paper)15.2656.6149.31

CASP has the highest average accuracy among the allocation methods shown at both targets and the lowest perplexity at 50%. Against GISP, CASP improves both perplexity and average accuracy at both targets. We have added these methods to Related Work. Once the remaining pruning ratios are evaluated, we will extend the allocation ablation table and add GISP to the main LLaMA2-7B results table.

AuthorsOpenReview ↗

Rebuttal for Reviewer TZbg (2/2)

3. Recent models and broader tasks.

Table 1.3 reports Qwen3-8B-Base results at 20% and 50% target pruning, including MMLU accuracy and GSM8K strict exact match.

Table 1.3. Qwen3-8B-Base results on language modeling and reasoning tasks.

TargetMethodWikiText2 PPL ↓PTB PPL ↓7-task avg. (%) ↑MMLU (%) ↑GSM8K (%) ↑
0%Dense6.9911.9670.4376.8485.15
20%SlimGPT9.1324.4563.9258.662.88
20%GISP10.5720.3957.5365.505.30
20%FLAP9.6625.8460.8761.320.00
20%CASP7.9615.7164.5667.065.76
50%SlimGPT18.57121.9942.9824.460.00
50%GISP133.49198.2637.6625.020.00
50%FLAP187.77995.6135.6023.520.00
50%CASP15.6542.3143.6526.520.30

Among the pruned models in Table 1.3, CASP achieves the lowest WikiText2 and PTB perplexities and the highest seven-task average accuracy, MMLU accuracy, and GSM8K accuracy at both pruning ratios. GSM8K scores drop sharply for all evaluated pruning methods, including CASP. They fall from 85.15% for the dense model to at most 5.76% at 20% pruning and 0.30% at 50%. Liu and Liu [1] suggest that generation errors can propagate across steps, making multi-step reasoning particularly sensitive to pruning. Diagnosing such errors and developing pruning criteria that better preserve reasoning are directions for future work. We will add the Qwen3-8B results to the existing additional-model appendix section once the remaining pruning ratios are evaluated.

[1] Songtao Liu and Peng Liu. 2026. High-Layer Attention Pruning with Rescaling. Transactions on Machine Learning Research.

4. Protected Uniform and allocation overhead.

Table 1.4 reports wall-clock time and peak GPU memory for sparsity allocation on LLaMA2-7B. CASP's peak CPU RSS for the full pruning pipeline is 1.75 GiB at 20% pruning and 1.76 GiB at 50% (Table 2.1).

Table 1.4. Sparsity allocation time and peak GPU memory on LLaMA2-7B.

TargetAllocationTime (s) ↓Peak GPU memory (GiB) ↓
20%LSA153.2518.74
20%OWL37.4518.74
20%Protected Uniform0.000130
20%CASP25.7125.30
50%LSA153.2518.74
50%OWL37.4518.74
50%Protected Uniform0.000150
50%CASP25.7125.30

CASP completes allocation in 25.71 s, faster than OWL and LSA and second only to Protected Uniform, which simply assigns a fixed schedule without loading the model. CASP's 25.30 GiB GPU peak includes the backward pass used to estimate layer sensitivity, a one-time offline cost that gradient checkpointing could reduce. Compared with Protected Uniform, CASP achieves lower perplexity on both WikiText2 and PTB at both pruning ratios (Table 1.1). Average accuracy rises from 61.89% to 63.27% at 20% pruning and from 40.51% to 49.31% at 50%. These gains make the one-time allocation cost a favorable trade-off, especially at 50% pruning. We will add the full allocation overhead comparison to a new appendix section once the remaining pruning ratios are evaluated.

Thank you for clarifying how CASP builds on LSA and for running the additional experiments. The new baseline comparisons on LLaMA2-7B and the MMLU and GSM8K results on Qwen3-8B-Base address my concern about the breadth of the evaluation. These results give me more confidence that CASP outperforms the baselines considered at both pruning ratios tested. Based on this evidence, I will raise my score from 2 to 3.

AuthorsOpenReview ↗

Response to Reviewer TZbg’s Follow-up Comments

Dear Reviewer TZbg,

Thank you for your valuable suggestions on CASP’s relationship to LSA and evaluation breadth, and for recognizing our additional comparisons and reasoning-task experiments. Your feedback has helped clarify our contribution and strengthen the evaluation. We will complete the remaining experiments and incorporate the results and analyses into the revised manuscript.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Discussion summary for AC

Dear Area Chairs,

Thank you for overseeing the review process. We sincerely appreciate the reviewers’ valuable feedback and would like to briefly summarize our rebuttal and discussion.

During the rebuttal, we have worked to revise the manuscript, strengthen the experimental evaluation, and provide detailed responses to the reviewers’ comments. These efforts focused on three areas:

  1. Scope of the contribution: We clarified CASP’s distinctions from existing work, including LSA and Kwon et al., and more precisely defined its contributions in structured interaction-aware pruning and gradient-based sparsity allocation.
  2. Additional experiments: We expanded baseline comparisons, model and task coverage, cost measurements, matched-budget evaluations, and GQA efficiency experiments.
  3. Technical clarifications: We clarified the reconstruction-error calculation, attention-head ablation, GQA implementation, and treatment of weight compensation.

We have responded to all reviewers’ comments. We believe these revisions strengthen the experimental evidence, clarify our contribution, and bring the paper to a standard suitable for acceptance.

Reviewers TZbg and FHKV acknowledged our efforts and indicated increases in their ratings from 2 to 3. We sincerely appreciate their recognition. We respectfully note that FHKV explicitly stated this increase in the follow-up discussion, although their Official Review still displays 2. We would appreciate this updated assessment being considered in the final evaluation.

Although Reviewer dHWe has not yet provided further feedback, we hope our detailed clarifications and additional experiments adequately address their concerns.

We hope our rebuttal and the follow-up discussion provide a helpful basis for your final assessment. Thank you again for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.