PRISM: Policy-driven Retraining-free Inter/Intra-layer Structured Merging for Accurate and Efficient LLM Pruning

Large language models (LLMs) incur substantial computation, memory, and latency costs during inference, motivating structured pruning for efficient deployment. However, directly removing layers or channels discards learned representations, often requiring fine-tuning or reconstruction to recover accuracy. Inter-layer pruning offers substantial computational savings at the risk of severe accuracy degradation, whereas intra-layer pruning provides finer-grained compression with more limited end-to-end acceleration. We introduce PRISM, a policy-driven, retraining-free framework that unifies inter- and intra-layer structured compression through weight merging. Rather than simply deleting redundant structures, PRISM transfers their information into retained weights, mitigating representation loss without additional training. For inter-layer compression, MHA-Removed FFN Composition removes selected multi-head attention (MHA) layers and composes adjacent feed-forward network (FFN) layers to exploit redundancy across layers. For intra-layer compression, Structure-Aware Channel Reconstruction merges channels while accounting for the distinct structural characteristics of MHA and FFN modules. To coordinate both compression levels, an Activation Distortion Score captures directional and magnitude changes in activations and guides the joint selection of inter-layer merging locations and layer-wise intra-layer sparsity. At 30% sparsity, PRISM achieves 1.25% lower perplexity than a state-of-the-art structured pruning baseline, while reducing inference computation and memory usage to 72.9% and 70.5% of the dense baseline, respectively. These results demonstrate that combining structured information transfer with activation-aware sparsity allocation improves the accuracy-efficiency trade-off of LLM compression without retraining.