- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Correlation-Aware Structured Pruning (CASP), a post-training method that explicitly models dependencies between prunable units in large language models. It formulates structured pruning as a cardinality-constrained binary quadratic program by decomposing the reconstruction error into individual costs and pairwise interaction terms, then solves it using an efficient greedy interaction algorithm with gradient-based layer-wise sparsity allocation. Experiments on multiple mainstream LLMs demonstrate that CASP achieves competitive accuracy–efficiency trade-offs, particularly under high pruning ratios.
Critical strengths
This paper's greedy interaction algorithm is computationally efficient, naturally supports both FFN channel pruning and attention-head pruning, and is complemented by an adaptive layer-wise sparsity allocation strategy. The experimental evaluation is comprehensive, covering multiple models, pruning ratios, benchmarks, and ablation studies that validate the contribution of each component.
Critical weaknesses
-
The method proposed in this paper still falls within the main line of post-training structured pruning, and the only difference from OSSCAR/SlimGPT/LLM-Pruner is that it explicitly formalizes the implicit dependencies and solves them better. Therefore, the innovation of the method in this paper is a major issue.
-
In the paper, OSSCAR jointly optimizes the structure set and the retained weights; CASP is post-training pruning, constructing Q and only selecting the masks without updating the retained weights. However, the distinction between implicit and explicit in the paper is not sufficient. When comparing the same reconstruction targets, the incremental effect relative to the existing related perception methods is not sufficiently separated. Therefore, the actual method effect needs to be further clarified.
-
In the GQA experiment of the paper, the pruning rules, pruning efficiency and model performance are reported, but the actual structure after pruning and the hardware efficiency after pruning are not given. There is no report of actual parameter quantity, the number of qkv per layer, and the comparison should be made fairly with other methods under the same actual parameter quantity or computing budget.
-
Table 3 shows that the budget allocation between layers will affect the final effect. It should be fixed at the same layer-wise budget to compare CASP, SlimGPT, and OSSCAR again, to distinguish the contributions of the interaction selection and allocation strategies.
-
In Appendix Figure 2, the curves are presented as continuous relaxation lower bounds, but the specific calculation method of the reconstruction error and the normalization method are not explained. Whether the same Q is used for each pruning rate.
-
In the experiments in Table 2 when pruning the attention heads, it is unclear how they are operated specifically: If the channel-level Q is diagonalized first, the interaction within the same head will also be deleted.
Questions for authors
See Weakness.
3 comments
Rebuttal for Reviewer dHWe (2/2)
4. Shared per-layer budgets.
On LLaMA2-7B, SlimGPT and OSSCAR use CASP's per-layer attention-head and FFN-channel budgets while retaining their own selection and weight compensation procedures. In Table 3.3, the rows marked “Original” show the baselines under their original budgets. These results and the CASP results are from Table 1 of the submitted paper. Accuracy is averaged over seven tasks.
Table 3.3. Pruning results under matched per-layer budgets on LLaMA2-7B.
| Target | Method | WikiText2 PPL ↓ | PTB PPL ↓ | Avg. accuracy (%) ↑ |
|---|---|---|---|---|
| 20% | SlimGPT (Original) | 7.29 | 29.68 | 63.01 |
| 20% | SlimGPT | 6.84 | 33.30 | 63.24 |
| 20% | OSSCAR (Original) | 7.03 | 26.67 | 63.08 |
| 20% | OSSCAR | 6.97 | 35.63 | 63.04 |
| 20% | CASP | 6.70 | 24.52 | 63.27 |
| 50% | SlimGPT (Original) | 18.36 | 62.36 | 48.04 |
| 50% | SlimGPT | 16.61 | 63.13 | 49.04 |
| 50% | OSSCAR (Original) | 20.43 | 60.97 | 35.48 |
| 50% | OSSCAR | 18.41 | 76.06 | 47.82 |
| 50% | CASP | 15.26 | 56.61 | 49.31 |
First, at 50% pruning, CASP's sparsity allocation improves average accuracy for both SlimGPT and OSSCAR, from 48.04% to 49.04% and from 35.48% to 47.82%, respectively (Table 3.3). Second, under the same per-layer budgets, CASP's interaction-aware pruning achieves the lowest PTB perplexity at 20% pruning and the best results on all three metrics at 50%. We will report the full matched-budget comparison in a new appendix section once the remaining pruning ratios are evaluated.
5. Figure 2 calculation and normalization.
All sparsities use the same from the first FFN layer of LLaMA2-7B. Each mask is scored by , the mean squared Frobenius reconstruction error over calibration samples, without further normalization. The lower bound minimizes the same objective over the continuous relaxation. We have clarified how the error is calculated in Appendix A.1 (p. 13) and specified the error scale in the Figure 2 caption (p. 14).
6. Attention-head Diag ablation.
Diag takes the diagonal after aggregating channel interactions by head (Section 4.2). This preserves within-head interactions and removes only between-head interactions. We have clarified this in the Interaction-Aware Selection paragraph of Section 5.3 (p. 9).
Follow-up on Our Rebuttal to Reviewer dHWe
Dear Reviewer dHWe,
Thank you for your valuable feedback on CASP’s methodological contribution and experimental evaluation. We have clarified the method’s contribution and relationship to prior work, and added experiments under matched per-layer budgets and on GQA hardware efficiency. We believe these changes clarify our contribution and strengthen the experimental evidence. With the discussion period ending on November 18, 2026 (AoE), we would greatly appreciate your thoughts on whether our responses address your concerns.
Rebuttal for Reviewer dHWe (1/2)
Dear Reviewer dHWe,
Thank you for identifying where the current comparisons mix allocation, mask selection, and weight recovery. We address these separately below.
1. Scope of the contribution.
Thank you for raising this important question, which goes to the core of our contribution. We respectfully clarify that the premise of a single difference does not hold: CASP is not a point-wise reformulation but a tightly coupled pipeline of three components.
At 50% pruning, CASP reaches 49.31% average accuracy, compared with 44.33% for Diag in the selection ablation and 45.17% for the strongest alternative schedule in the allocation ablation (Tables 2 and 3 of the submitted paper). With matched per-layer budgets, CASP also achieves a PTB perplexity of 56.61, compared with 63.13 for SlimGPT and 76.06 for OSSCAR (Table 3.3). Together, these results show the value of combining interaction-aware selection with allocation based on end-to-end loss, particularly at higher sparsity.
2. Weight compensation.
OSSCAR jointly optimizes the mask and retained weights, and CASP selects the mask by explicitly modeling pairwise interactions in the fixed-weight reconstruction objective. Refitting weights can lower local reconstruction error, with a risk of overfitting on a small calibration set.
Table 3.1. CASP and OSSCAR under matched per-layer budgets on LLaMA2-7B.
Table 3.1 shows that under matched per-layer budgets, CASP outperforms OSSCAR on all three metrics at both pruning ratios, lowering PTB perplexity by 31.2% at 20% pruning and 25.6% at 50%.
In addition, as reported in Appendix B.6, LoRA fine-tuning on additional training data raises CASP's average accuracy from 49.31% to 53.43% at 50% pruning. This approach may offer more robust recovery than weight compensation on a small calibration set.
3. GQA structure and hardware efficiency.
Table 3.2 reports parameter counts and inference measurements for physically pruned LLaMA3-8B-Base models at comparable parameter counts. CASP prunes query heads and removes a shared key/value head only when all its associated query heads have been pruned. The target column denotes CASP’s nominal pruning ratio. We report MACs for batch size 1 and sequence length 128.
Table 3.2. Retained attention heads and inference efficiency on LLaMA3-8B-Base.
Head counts are summed across layers. LoRAP uses low-rank compression and preserves the original head counts.
At both pruning targets, the methods have comparable parameter counts and MACs (Table 3.2). CASP's prefill and decode latencies and peak GPU memory are close to LLM-Pruner's. Compared with LoRAP, CASP records lower prefill and decode latencies at both targets, with nearly identical peak GPU memory at 20% and lower peak memory at 50%. At the 50% target, CASP achieves the lowest perplexity on WikiText2 and PTB (33.25 and 53.03) and the highest average zero-shot accuracy (45.20%, versus 40.60% for LLM-Pruner and 43.93% for LoRAP), as reported in Table 10 of the submitted paper. These results show that CASP retains higher model quality while keeping inference costs competitive. We will report the full GQA hardware comparison in a new appendix section once the remaining pruning ratios are evaluated.