acceptodds
Under review as a conference paper at ICLR 2027

PRISM: Policy-driven Retraining-free Inter/Intra-layer Structured Merging for Accurate and Efficient LLM Pruning

Abstract

Large language models (LLMs) incur substantial computation, memory, and latency costs during inference, motivating structured pruning for efficient deployment. However, directly removing layers or channels discards learned representations, often requiring fine-tuning or reconstruction to recover accuracy. Inter-layer pruning offers substantial computational savings at the risk of severe accuracy degradation, whereas intra-layer pruning provides finer-grained compression with more limited end-to-end acceleration. We introduce PRISM, a policy-driven, retraining-free framework that unifies inter- and intra-layer structured compression through weight merging. Rather than simply deleting redundant structures, PRISM transfers their information into retained weights, mitigating representation loss without additional training. For inter-layer compression, MHA-Removed FFN Composition removes selected multi-head attention (MHA) layers and composes adjacent feed-forward network (FFN) layers to exploit redundancy across layers. For intra-layer compression, Structure-Aware Channel Reconstruction merges channels while accounting for the distinct structural characteristics of MHA and FFN modules. To coordinate both compression levels, an Activation Distortion Score captures directional and magnitude changes in activations and guides the joint selection of inter-layer merging locations and layer-wise intra-layer sparsity. At 30% sparsity, PRISM achieves 1.25% lower perplexity than a state-of-the-art structured pruning baseline, while reducing inference computation and memory usage to 72.9% and 70.5% of the dense baseline, respectively. These results demonstrate that combining structured information transfer with activation-aware sparsity allocation improves the accuracy-efficiency trade-off of LLM compression without retraining.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large language models, retraining-free compression, structured pruning, structured weight merging, inter- and intra-layer compression, activation-aware sparsity allocation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ZzHROpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes PRISM, a structured compression framework that combines depth reduction and channel compression without gradient-based retraining. It removes selected attention modules, composes adjacent FFNs, and reconstructs compressed channels to transfer information into retained weights. An activation-distortion score guides where and how much to compress. Experiments on several LLM families show competitive perplexity and downstream accuracy, including extensions to GQA models and models up to 32B. Measurements on LLaMA2-7B further report generation latency, memory use, and the compression pipeline's wall-clock cost.

Critical strengths
  1. Useful compression pipeline with broader model validation. PRISM combines inter-layer composition and intra-layer reconstruction, and Appendix F extends the main experiments to LLaMA-3.1 and Qwen3, including 32B. On Qwen3-32B, it reaches 66.10 average accuracy versus Olica's 64.53 at approximately 30% sparsity. This supports applicability beyond the older 7B models in the main table and shows that the benefit can extend to a substantially larger GQA model.
  2. Measured quality–efficiency trade-offs. Tables 2–3 report actual generation latency and compression-stage timing rather than relying only on parameter reduction. At 39% sparsity, PRISM improves Olica's PPL from 28.14 to 24.98 and latency from 5.13 to 4.90 seconds on the measured model. This provides direct evidence of a better operating point in that workload, and the stage breakdown identifies activation-distortion measurement as the main offline cost.
Critical weaknesses
  1. End-to-end component contributions are not isolated. Table 4 ablates the allocation metric, while supplementary Figure 6 evaluates MRFC/SACR using local activation distortion. These do not show how much each merging module contributes to final PPL, downstream accuracy, or latency at a matched compression budget. Because ADS is also the selection criterion, improving local ADS alone does not establish that both modules are necessary for the end-task gain. Representative matched-budget module ablations would strengthen the central combination claim.
  2. The practical efficiency advantage is difficult to assess or reproduce. Detailed timings cover only LLaMA2-7B generating 256 tokens; at 30% sparsity the Olica–PRISM latency difference is 5.71 versus 5.62 seconds. The manuscript does not specify the timing GPU, precision, batch size, input length, or repeated-run variability. These details are particularly important for a difference of about 1.6%, and their absence leaves both its stability and the GQA/larger-model efficiency benefit unverified.
Questions for authors
  1. Are existing matched-budget results available for MRFC-only, SACR-only, and their combination, using final task metrics rather than local ADS? If not, please qualify the component-level conclusions.
  2. Please specify the complete timing protocol for Tables 2–3/8, including hardware, precision, batch/input sizes, warm-up, and repeats. Is the 0.09-second advantage at 30% sparsity stable across runs?
5 comments

Response to Reviewer ZzHR (1/3): What do SACR and MRFC contribute at matched sparsity?

Thank you for asking for end-to-end evidence of each component’s contribution and for a measurement protocol that puts the reported 0.09-second latency difference in context. We address the two points separately below.

W1/Q1. At a matched compression budget, what do MRFC and SACR each contribute to final model quality?

  • How we isolated the operators. On LLaMA2-7B at 30% sparsity, we fixed the removal locations and allocation. Changing only the FFN operator tests whether SACR improves on channel removal; changing only the inter-layer operator tests the effect of composing adjacent FFNs. We report final PPL and seven-task accuracy rather than local activation distortion.

Table ZzHR-1. Operator ablations at a matched budget. PPL is averaged over three calibration sets. Accuracy is the seven-task mean for the model compressed with the manuscript calibration set; all rows have 4.7396B parameters on that set.

Inter-layer operatorFFN operatorPPL-128PPL-2048Accuracy (%)
MRFCRemoval20.4910.2957.75
MRFCSACR18.178.2759.99
Attention DropRemoval19.729.9858.23
Attention DropSACR17.417.9760.23
Unaligned concatSACR17.698.0960.16
Input-aligned concat onlySACR17.818.1460.01
  • SACR improved quality with either inter-layer operator. With MRFC fixed, replacing FFN channel removal with SACR lowered PPL-128 from 20.49 to 18.17 and raised accuracy by 2.24 percentage points. With Attention Drop fixed, PPL-128 fell from 19.72 to 17.41 and accuracy rose from 58.23% to 60.23% (2.00 points).
  • MRFC reduced depth, with a quality cost at the default fit. With SACR fixed, MRFC used 30 rather than 32 blocks and scored 0.24 percentage points below Attention Drop in accuracy. Unaligned concatenation, also at 30 blocks, had lower PPL than the default MRFC fit. We therefore varied the amount of data used to fit MRFC without changing the compressed architecture.

Table ZzHR-2(a). LLaMA2-7B: varying only the ridge-fitting sample count. Unlike the three-set means in Table ZzHR-1, this experiment fixes the removal locations, allocation, and the calibration data for all other steps (128 BookCorpus samples). The MRFC regressions use dense-model inputs.

ConfigurationBlocksRidge samplesPPL-128PPL-2048
Attention Drop32N/A17.417.94
Unaligned concat30N/A17.658.06
MRFC30C4 12818.038.22
MRFC30C4 51217.718.08
MRFC30C4 2,04817.668.06
MRFC30C4 8,19217.638.05
  • Two fewer blocks with a smaller PPL gap. Increasing C4 from 128 to 512 fitting samples reduced the PPL-128 gap to Attention Drop from 0.62 to 0.30. At 8,192 samples the gap was 0.22; the PPL-2048 gap likewise fell from 0.28 to 0.11. At that point MRFC and the equally deep unaligned-concatenation model differed by only 0.02 and 0.01 PPL at the two lengths.

Table ZzHR-2(b). Inter-layer operators on four additional models at 30% sparsity. Each method uses the same calibration set within a model (seed 0). Values are WikiText-2 PPL at lengths 128 / 2,048. Each Δ is MRFC PPL-128 minus the respective baseline PPL-128.

ModelAttention DropUnaligned concatMRFCΔ vs. concatΔ vs. Drop
LLaMA-3.1-8B35.07 / 15.1936.49 / 15.5939.20 / 16.66+2.71+4.13
Qwen3-8B75.00 / 34.3674.93 / 34.5475.93 / 34.82+1.00+0.93
LLaMA2-13B14.70 / 6.5414.76 / 6.5715.00 / 6.66+0.24+0.30
Qwen3-32B28.88 / 12.1129.22 / 12.2430.10 / 12.52+0.88+1.22
  • Default MRFC across models. Both controls had lower PPL than MRFC at both lengths on all four models. The PPL-128 gap to Attention Drop ranged from 0.30 on LLaMA2-13B to 4.13 on LLaMA-3.1-8B. To distinguish sensitivity to the ridge fit from the effect of the architecture, we kept each model’s structure and allocation fixed and varied only the fitting data.

Response to Reviewer ZzHR (2/3): How do calibration, block count, and allocation affect the results?

W1/Q1 (continued). Does fitting MRFC on more samples narrow the quality gap to Attention Drop, and what do we gain from reducing depth?

Table ZzHR-2(c). MRFC PPL-128 / PPL-2048 as a function of ridge-fitting sample count on the four additional models. NN denotes fitting samples; a dash denotes an unreported result.

ModelRidge dataN=128N=512N=2,048N=8,192
LLaMA-3.1-8BBookCorpus40.03 / 16.8636.46 / 15.7036.30 / 15.6136.24 / 15.58
LLaMA-3.1-8BC439.62 / 16.6836.24 / 15.5436.03 / 15.4835.97 / 15.46
Qwen3-8BBookCorpus79.75 / 35.9073.45 / 34.2073.01 / 33.9872.70 / 33.87
Qwen3-8BC478.63 / 35.0472.55 / 33.4872.15 / 33.3572.10 / 33.35
LLaMA2-13BBookCorpus14.94 / 6.6414.80 / 6.5814.75 / 6.57—
LLaMA2-13BC414.98 / 6.6514.77 / 6.5714.75 / 6.57—
Qwen3-32BBookCorpus30.33 / 12.5929.51 / 12.3429.24 / 12.25—
Qwen3-32BC429.88 / 12.4629.39 / 12.2929.12 / 12.21—
  • Effect of more fitting data. Increasing either BookCorpus or C4 from 128 to 2,048 samples lowered both PPLs on all four models. On Qwen3-8B, 512 C4 samples gave 72.55 / 33.48, below both Attention Drop (75.00 / 34.36) and unaligned concatenation (74.93 / 34.54). On LLaMA-3.1-8B, 8,192 C4 samples put MRFC below concatenation but above Attention Drop. On LLaMA2-13B and Qwen3-32B, the remaining gaps to Attention Drop at 2,048 C4 samples were 0.05 / 0.03 and 0.24 / 0.10, respectively.

Table ZzHR-2(d). Time to generate 256 tokens with each inter-layer operator on LLaMA2-7B at 30% sparsity. We used Hugging Face’s eager generate on one RTX 4090 (FP16, batch size 1). Values are means ± standard deviations across five sessions.

Inter-layer operatorBlocks128-token prompt (s)1,024-token prompt (s)
MRFC306.435 ± 0.1116.434 ± 0.193
Attention Drop326.533 ± 0.0746.689 ± 0.107
Unaligned concat306.322 ± 0.2706.449 ± 0.175
  • Fewer blocks, lower generation latency. Attention Drop leaves the two neighboring FFNs in sequence. MRFC composes them, reducing the number of blocks executed from 32 to 30. MRFC took 0.098 s less (1.5%) than Attention Drop on average with a 128-token prompt and 0.255 s less (3.8%) with a 1,024-token prompt. Unaligned concat, which also executes 30 blocks, was faster than Attention Drop at both prompt lengths.

  • Component result versus full-model result. On LLaMA2-7B, MRFC trades two blocks for a higher default PPL than Attention Drop; increasing the ridge sample count narrows that gap without adding the blocks back (Table ZzHR-2(a)). In the separate comparison of complete methods in manuscript Table 2, PRISM reached 18.16 PPL-128 at 30% sparsity, compared with 18.39 for Olica and 30.70 for CoMe. Those manuscript values do not use the best result from the ridge sweep.

Table ZzHR-3. Changing allocation while keeping the operators fixed. PPL and accuracy use the same aggregation as Table ZzHR-1. Within each operator family, the parameter counts differ by approximately 0.10–0.48% between allocation strategies.

OperatorsAllocationParameters (B)PPL-128PPL-2048Accuracy (%)
OlicaUniform4.725018.498.1259.37
OlicaADS/SAP4.729818.077.9760.10
PRISMUniform4.716819.298.5559.23
PRISMADS/SAP4.739618.178.2759.99
  • Allocation has a separate effect. Switching from uniform allocation to ADS/SAP lowered PPL at both lengths for both operator families. Accuracy rose by 0.73 percentage points for Olica and by 0.76 for PRISM with MRFC and SACR. Olica + ADS/SAP performed best in this comparison. In the revision, we will distinguish this allocation effect from SACR’s gain over channel removal and from MRFC’s depth–quality trade-off.

Response to Reviewer ZzHR (3/3): Does PRISM’s latency advantage over Olica persist across runs?

W2/Q2. How was latency measured, and does the original 0.09-second advantage persist across runs?

  • Original measurement. Manuscript Tables 2 and 8 report inference time including prompt processing and generation of 256 tokens. At 30% sparsity, Olica took 5.71 s and PRISM took 5.62 s, averaged over 10 runs on one RTX 4090, LLaMA2-7B, FP16, batch size 1, with a 128-token input. We retained the means but not the individual timings, so we cannot recover the variance of that original difference. Manuscript Table 3 instead reports the offline, per-stage cost of compressing one model.
  • Repeat protocol. On the same device and at the same precision, we used Hugging Face’s eager generate with 128- and 1,024-token inputs and a fixed 256-token generation length. We randomized model order within each session and ran three warm-ups followed by ten timed runs per model. The speedup reference in Table ZzHR-4 is the effectively unpruned Olica implementation, which uses the same execution path as the compressed models.

Table ZzHR-4. Generation latency and speedup relative to the Olica Dense reference at two input lengths.

Model128-token input: latency (s)Speedup vs. reference1,024-token input: latency (s)Speedup vs. reference
Olica Dense, nominal SR 0%6.683 ± 0.1241.000×6.824 ± 0.1281.000×
Olica, SR 30%6.469 ± 0.0991.033 ± 0.025×6.583 ± 0.0471.037 ± 0.021×
PRISM, SR 30%6.185 ± 0.2121.080 ± 0.042×6.270 ± 0.0511.088 ± 0.022×
  • Direct comparison with Olica at 30% sparsity. Using the mean latencies in the table, PRISM took 0.284 s less (4.39%) at 128 input tokens and 0.313 s less (4.75%) at 1,024. In all three sessions that included both compressed models, PRISM also had the lower median latency at both input lengths.
  • What was reproducible? For each session, we divided Olica’s median over ten runs by PRISM’s median. The mean and between-session standard deviation of that ratio were 1.051 ± 0.015× at 128 input tokens and 1.060 ± 0.011× at 1,024. These measurements reproduce the direction of the advantage over Olica under this workload; they do not establish that the original difference is always exactly 0.09 s.
Reviewer_MfA1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes PRISM, a retraining-free structured compression framework for LLMs. It combines inter-layer (depth) and intra-layer (width) compression and replaces direct removal with weight merging. It has three components:

  • MHA-Removed FFN Composition (MRFC). MRFC drops selected attention layers. It aligns the input of the following FFN to the preceding FFN’s input with a ridge-regression transform TXT_X, which is absorbed into the gate/up weights. It then re-fits the down projection and concatenates the two FFNs into one.

  • Structure-Aware Channel Reconstruction (SACR). SACR applies truncated SVD to WqW_q, WkW_k and the product WvWoW_v W_o. In the FFN it keeps salient channels and merges redundant ones into a few merged channels, using pivot-based TGT_G, TUT_U and a ridge-regression down projection.

  • Activation Distortion Score (ADS). ADS is a max-normalized sum of cosine distance and relative L2 error. A sparsity allocation policy uses it to choose MRFC locations and per-layer QK/VO/FFN ratios under a target budget.

Experiments cover LLaMA-1/2-7B, Vicuna-7B, LLaMA-2-13B, LLaMA-3.1-8B, Qwen3-8B/32B at 21–39% sparsity. They report WikiText-2 PPL (sequence length 128), seven zero-shot tasks, latency and memory on LLaMA2-7B, a pipeline time breakdown, and ablations. PRISM generally achieves the best PPL and average accuracy among retraining-free baselines (ShortGPT, SLEB, CoMe, LoRAP, Olica). It is also slightly faster than the intra-layer baseline Olica.

Critical strengths
  1. Practical retraining-free hybrid compression with real depth reduction. Intra-layer methods like Olica keep depth and give little speedup (around 1.0–1.2×). Inter-layer methods like CoMe are fast but lose much quality. PRISM gets both slightly lower latency and lower PPL than Olica at matched budgets, with no gradient-based training and about 15 minutes of compression time on a 7B model. This latency–quality trade-off without retraining is the main reason the paper is of practical interest.

  2. Broad evaluation and informative ablations. The evaluation covers several model families, including GQA models (LLaMA-3.1, Qwen3) and scales up to 32B. It also includes a limited-budget comparison with Minitron. The component analyses support the design choices at the level of activation distortion.

Critical weaknesses
  • Several core novelty claims are overstated in comparison to established literature. Specifically, the framework introduces “We reformulate structured pruning as structured weight merging” (Contribution 2) as a primary innovation; however, merging-based compression paradigms (e.g., CoMe, FFN Fusion, and LEMON) are already well-documented and recognized within the paper itself. This contradicts the premise that existing methods rely solely on permanent parameter removal.

  • The core premise of MRFC, that Multi-Head Attention (MHA) modules can be pruned more safely than Feed-Forward Network (FFN) modules (Appendix D, Fig. 5), is presented as an original insight without citing relevant prior literature, such as He et al. (2024, What Matters in Transformers? Not All Attention is Needed). Consequently, the evaluation lacks a critical comparison against a standard Attention Drop baseline to demonstrate that the proposed FFN composition yields measurable gains over straightforward attention pruning at equivalent compute budgets.

  • The allocation policy is described as "principled," yet the Activation Distortion Score (ADS) relies on an empirical summation of cosine distance and relative L2 error weighted by hyperparameter 1 / max. Because its validation is strictly qualitative and limited to a single architecture (Fig. 2), the formulation functions more as a targeted heuristic than a broadly validated theoretical objective.

  • Significant performance improvements are primarily observed under aggressive compression regimes (39% sparsity) on specific models (Qwen3 and LLaMA-3.1). Under these conditions, several baseline behaviors (such as Olica exhibiting a perplexity spike to 222 on Qwen3-8B) raise significant concerns regarding evaluation fairness and comparative baseline robustness.

  • The empirical evaluation does not isolate whether performance gains stem from the ADS allocation strategy or the weight merging operators themselves. Essential controlled experiments, such as evaluating baseline methods using ADS allocation or applying uniform allocation to PRISM operators, are omitted, making it difficult to attribute performance improvements to specific architectural choices.

Questions for authors
  1. How exactly does SACR’s MHA compression differ from Olica’s decomposition of WqWk⊤W_q W_k^\top and WvWoW_v W_o, and from MoDeGPT? Could you report a variant of PRISM that uses Olica’s MHA module, to isolate your contribution?

  2. Can you compare MRFC with (a) attention removal alone (Attention Drop), (b) FFN Fusion-style concatenation without alignment, and (c) a linear-compensation baseline such as ReplaceMe, at matched sparsity and in end-to-end PPL and accuracy, not only layer-wise ADS?

  3. Could you disentangle allocation from operators? For example: PRISM operators with uniform allocation, and Olica with ADS-based allocation.

  4. How were baselines tuned on GQA models, in particular Olica on Qwen3-8B (PPL 222.33)? Is the GQA-to-MHA unfolding applied identically to all methods, and does it change the actual KV-cache size of the final model?

  5. Why is WikiText-2 PPL evaluated at sequence length 128 rather than the more common 2048? Do the rankings hold at longer contexts? This seems especially relevant because MRFC removes entire attention layers.

6 comments

Response to Reviewer MfA1 (1/4): What is new in PRISM, and what does MRFC add on LLaMA2-7B?

Thank you for asking us to distinguish the contribution of each merging operator from that of the allocation policy. We address these questions with matched-budget operator comparisons, a ridge-calibration study, and tests of ADS on held-out text.

W1. What does PRISM contribute beyond earlier work on weight merging?

  • Prior work. We agree that merging itself is not new. CoMe selects and connects important channels, FFN Fusion seeks to preserve FFN function during fusion, and LEMON studies fusion across layers and dimensions. We will revise the statement that characterized previous methods collectively as direct removal.
  • Our contribution. PRISM addresses two decisions: how to retain information from structures selected for compression, and where to apply each operation. SACR leaves salient channels intact and reconstructs the output contribution of lower-importance channels with a small set of merged channels. MRFC composes the two FFNs left after attention removal. ADS/SAP uses the resulting activation distortion to choose compression locations and amounts. In W5/Q3, we test reconstruction at fixed allocation and allocation with the operators held fixed.

W2/Q2. What does MRFC add over Attention Drop, FFN concatenation, or linear compensation?

  • Why compose the FFNs? He et al. (2024), What Matters in Transformers? Not All Attention is Needed, already observed that some attention modules can be removed. We will credit this observation. After attention removal, two neighboring FFNs still execute in sequence. MRFC composes them into one FFN, reducing the number of blocks the model must execute.
  • Operator comparison. On LLaMA2-7B at 30% sparsity, we held the attention-removal locations, layer-wise allocation, and SACR fixed. Unaligned concatenation uses only a diagonal correction based on the RMSNorm ratio. Input-aligned concatenation adds ridge alignment but does not refit the down projection. The ReplaceMe-style control absorbs a linear correction into the preceding FFN’s down projection.

Table MfA1-1(a). End-to-end performance with only the inter-layer operator changed. PPL is averaged over three calibration sets. Accuracy is the seven-task mean for the model compressed with the manuscript calibration set; all configurations have 4.7396B parameters on that set.

Inter-layer operator (SACR throughout)BlocksPPL-128PPL-2048Accuracy (%)
MRFC3018.178.2759.99
Attention Drop3217.417.9760.23
Unaligned concat3017.698.0960.16
Input-aligned concat only3017.818.1460.01
ReplaceMe-style compensation3217.458.0260.07
  • Depth and quality. Attention Drop gave the best PPL and accuracy at 32 blocks. MRFC used 30 blocks, with accuracy 0.24 percentage points lower (59.99% vs. 60.23%). At the default calibration size, unaligned concatenation also had lower PPL than MRFC at the same depth of 30 blocks.

Table MfA1-1(b). LLaMA2-7B: varying only the data used to fit MRFC’s ridge regressions. Unlike the three-set means in (a), this comparison fixes the removal locations, allocation, and all other calibration steps (128 BookCorpus samples). We use dense-model inputs for ridge fitting. The fixed PPL-128 / PPL-2048 references are 17.41 / 7.94 for Attention Drop and 17.65 / 8.06 for unaligned concatenation.

Ridge samplesBookCorpus: PPL-128 / PPL-2048C4: PPL-128 / PPL-2048
12818.24 / 8.2718.03 / 8.22
51217.77 / 8.1017.71 / 8.08
2,04817.72 / 8.0917.66 / 8.06
8,19217.69 / 8.0717.63 / 8.05
  • Effect of ridge sample count. Increasing C4 from 128 to 8,192 samples narrowed the MRFC–Attention Drop gap from 0.62 to 0.22 at length 128 and from 0.28 to 0.11 at length 2,048. With BookCorpus, the corresponding gaps fell from 0.83 to 0.28 and from 0.33 to 0.13. Attention Drop still had the lowest PPL; at 8,192 C4 samples, MRFC was within 0.02 and 0.01 PPL of the 30-block concatenation model.

We continue W2/Q2 in part 2 with results on four additional models and generation timings on LLaMA2-7B.

Response to Reviewer MfA1 (2/4): How does MRFC affect perplexity across models and generation time on LLaMA2-7B?

W2/Q2 (continued). Do the PPL results generalize, and does reducing depth lower generation latency?

Table MfA1-1(c). Inter-layer operators on four additional models at 30% sparsity. The three methods use the same calibration set within each model (seed 0). Values are WikiText-2 PPL at lengths 128 / 2,048. Each Δ is MRFC PPL-128 minus the respective baseline PPL-128.

ModelAttention DropUnaligned concatMRFCΔ vs. concatΔ vs. Drop
LLaMA-3.1-8B35.07 / 15.1936.49 / 15.5939.20 / 16.66+2.71+4.13
Qwen3-8B75.00 / 34.3674.93 / 34.5475.93 / 34.82+1.00+0.93
LLaMA2-13B14.70 / 6.5414.76 / 6.5715.00 / 6.66+0.24+0.30
Qwen3-32B28.88 / 12.1129.22 / 12.2430.10 / 12.52+0.88+1.22
  • Does the default result generalize? With the default fit, MRFC had higher PPL than both controls at both lengths on all four models. Its PPL-128 gap to Attention Drop ranged from 0.30 on LLaMA2-13B to 4.13 on LLaMA-3.1-8B. We therefore repeated the ridge-only sample sweep on each model.

Table MfA1-1(d). MRFC PPL-128 / PPL-2048 as a function of ridge-fitting sample count. NN denotes fitting samples; a dash denotes an unreported result.

ModelRidge dataN=128N=512N=2,048N=8,192
LLaMA-3.1-8BBookCorpus40.03 / 16.8636.46 / 15.7036.30 / 15.6136.24 / 15.58
LLaMA-3.1-8BC439.62 / 16.6836.24 / 15.5436.03 / 15.4835.97 / 15.46
Qwen3-8BBookCorpus79.75 / 35.9073.45 / 34.2073.01 / 33.9872.70 / 33.87
Qwen3-8BC478.63 / 35.0472.55 / 33.4872.15 / 33.3572.10 / 33.35
LLaMA2-13BBookCorpus14.94 / 6.6414.80 / 6.5814.75 / 6.57—
LLaMA2-13BC414.98 / 6.6514.77 / 6.5714.75 / 6.57—
Qwen3-32BBookCorpus30.33 / 12.5929.51 / 12.3429.24 / 12.25—
Qwen3-32BC429.88 / 12.4629.39 / 12.2929.12 / 12.21—
  • How much of the gap closes? On every model, increasing either BookCorpus or C4 from 128 to 2,048 samples lowered PPL at both evaluation lengths. On Qwen3-8B, 512 C4 samples gave 72.55 / 33.48, below both controls in (c). On LLaMA-3.1-8B, 8,192 C4 samples put MRFC below concatenation but still above Attention Drop. At 2,048 C4 samples, the remaining gaps to Attention Drop were 0.05 / 0.03 on LLaMA2-13B and 0.24 / 0.10 on Qwen3-32B.

Table MfA1-1(e). Time to generate 256 tokens with each inter-layer operator on LLaMA2-7B at 30% sparsity. We used Hugging Face’s eager generate on one RTX 4090 (FP16, batch size 1). Values are means ± standard deviations across five sessions.

Inter-layer operatorBlocks128-token prompt (s)1,024-token prompt (s)
MRFC306.435 ± 0.1116.434 ± 0.193
Attention Drop326.533 ± 0.0746.689 ± 0.107
Unaligned concat306.322 ± 0.2706.449 ± 0.175
  • Fewer blocks, lower generation latency. Attention Drop leaves the two neighboring FFNs in sequence. MRFC composes them, reducing the number of blocks executed from 32 to 30. MRFC took 0.098 s less (1.5%) than Attention Drop on average with a 128-token prompt and 0.255 s less (3.8%) with a 1,024-token prompt. Unaligned concat, which also executes 30 blocks, was faster than Attention Drop at both prompt lengths.

  • MRFC versus the full method. The LLaMA2-7B ablation shows what MRFC contributes: it reduces depth from 32 to 30 blocks, and a larger ridge fit narrows the quality gap without adding either block back. The manuscript’s Table 2 answers a separate question about complete compression methods: at 30% sparsity on LLaMA2-7B, PRISM achieved 18.16 PPL-128, compared with 18.39 for Olica and 30.70 for CoMe. These are the manuscript results, not the best values from the ridge sweep.

Response to Reviewer MfA1 (3/4): Does ADS predict held-out loss, and how was Olica evaluated on GQA models?

W3. Does ADS predict quality loss on data not used for calibration?

  • What ADS measures. ADS combines cosine distance and relative L2 error. We will describe it as an empirical criterion and report how well it ranks candidates by loss on a separate text set. A positive Spearman correlation means that candidates with lower ADS tend to have lower held-out loss, or a smaller increase in loss.

Table MfA1-2. Spearman rank correlation between ADS and held-out loss on LLaMA2-7B.

Candidate setCountQuantities comparedSpearman ρ
All individual operations320ADS vs. increase in held-out loss0.86
MHA removal locations32Removal ADS vs. increase in held-out loss0.68
Complete policies, fixed attention/FFN split16Total ADS vs. held-out loss0.635
  • Individual operations. In each of 32 layers, we tested QK, VO, and FFN compression at three retention levels (25.5%, 50%, 74.5%), plus removal of the MHA module: 32 × (3 × 3 + 1) = 320 candidates. Pooling these candidates tests whether ADS can compare different operations (ρ = 0.86). Restricting the analysis to the 32 MHA removals tests whether it distinguishes locations for the same operation (ρ = 0.68).
  • Complete policies. Across 16 policies with the attention/FFN budget split fixed and the removal locations varied, total ADS correlated with held-out loss (ρ = 0.635). This supports using ADS to rank removal locations at a given budget split.

W4/Q4. How was Olica tuned on GQA models, and what happens to the KV cache?

  • Architecture and tuning. Both methods retained native GQA without unfolding to MHA. Following Olica’s allocation rule, we specified QK and VO compression as multiples of the sparsity ratio and assigned the remaining budget to the FFN. Held-out BookCorpus loss selected (2, 0.25) rather than the original (2, 0.5) on both models: QK compression stayed fixed, while VO compression decreased. Our GQA adaptation omits Olica’s original FFN linear correction. The GQA implementation of PRISM also uses Olica-style channel selection for FFN width reduction.

Table MfA1-3. Native GQA at approximately 30% sparsity. The first PPL column is Olica as reported in our manuscript’s Table 6. The reevaluations use one common calibration set per model. KV counts are cache elements per token relative to the dense model.

ModelOlica PPL-128 (manuscript Table 6)Tuned Olica PPL-128PRISM PPL-128PPL-2048 (Olica / PRISM)KV elements vs. dense (PRISM / Olica)
LLaMA-3.1-8B44.6440.8239.2018.93 / 16.6689.0% / 92.6%
Qwen3-8B222.3163.575.9367.7 / 34.8293.3% / 92.6%
  • Matched comparison. On Qwen3-8B, Olica’s PPL-128 fell from the originally reported 222.3 to 163.5 after reevaluation and tuning; PRISM obtained 75.93 under the same reevaluation conditions. On LLaMA-3.1-8B, the corresponding values were 40.82 and 39.20. Both methods reduced KV storage relative to dense. On Qwen3-8B, PRISM retained 93.3% of the dense KV elements, slightly more than Olica’s 92.6%.
Reviewer_GPsyOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies training-free structured compression of LLMs by jointly considering inter-layer and intra-layer redundancy. It combines attention removal and FFN fusion with input alignment and output reconstruction and introduces structure-aware channel merging mechanism. Experiments across several model families and benchmarks report improvements in perplexity and competitive zero-shot performance over the compared baselines. It also shows promising latency reductions.

Critical strengths
  • The addresses the trade-off between model quality, compression ratio, and inference latency by combining depth and width compression.
  • It reports competitive performance across multiple model families and sizes. These results demonstrate the approach’s practical potential.
Critical weaknesses
  • The major concern I have is the clarity. Some core components are insufficiently specified. For example, TGT_G and TUT_U in SACR only described qualitatively without precise formulation. SAP also lacks sufficient detail regarding candidate compression ratios.
  • More analysis is needed to further validate the reliability. For example, does a lower ADS actually predict a lower loss on held-out text? An analysis like rank-correlation could check this for both individual operations and complete policies at the same sparsity. This would provide more direct evidence that ADS can reliably guide SAP’s choices.
Questions for authors

See above.

3 comments

Response to Reviewer GPsy

Thank you for identifying two points that need clarification: how SACR and SAP are implemented, and whether ADS predicts loss on text not used for calibration. We give the construction and the requested rank correlations below.

W1. How are TGT_G and TUT_U constructed and applied, and which policies does SAP search?

  • SACR: select the channels to merge. For each FFN intermediate channel, we sum the norms of its gate row, up row, and down-projection column, weighted by the corresponding calibration activation statistics. We leave high-importance channels unchanged and designate the lowest-ranked mm channels as the merge candidates (red\mathrm{red}). Within those candidates, the approximately 10% with the largest up-activation norms serve as pivots, one per merged channel.
  • Construct the two mixing matrices. TGT_G and TUT_U use the same pivots but different mixing weights: gate-activation norms for TGT_G, and up-activation norms for TUT_U. If hjh_j is the activation norm for candidate channel jj in the respective branch, let dj=1−hj/max⁡ihid_j=1-h_j/\max_i h_i. In each column of the mixing matrix, the pivot entry is 0.9; each nonpivot entry is dj/(10∑idi)d_j/(10\sum_i d_i). The gate and up projections therefore retain the same representative channel while incorporating different contributions from the remaining channels.
  • Apply the matrices and refit the output. The transposed mixing matrices act on the candidate rows of the gate and up projections:
W_{\mathrm{up}}^{\mathrm{mg}}=T_U^\top W_{\mathrm{up}}^{\mathrm{red}}.$$ - **Recover the output contribution.** We fit the down projection by ridge regression from the merged intermediate activations to **the original candidates’ contribution to the FFN output**. The final FFN comprises the untouched high-importance channels and these reconstructed channels. - **SAP: measure candidate operations.** On LLaMA2-7B, we evaluate QK, VO, and FFN retention at **41 levels from 1% to 99%** per layer. ADS measures distortion relative to the original block output. We separately score complete MHA removal and combine QK and VO into an attention-compression path using their measured distortions. - **Choose candidate removal locations.** From the seven layers with the lowest MHA-removal ADS, we enumerate subsets of **zero to four nonadjacent layers**. Composing neighboring FFNs changes their width and inputs, so we remeasure the composed FFN at 81 levels and reevaluate the corresponding attention path. - **Divide the budget and select the policy.** At 30% target sparsity, for example, the share of total model parameters removed from attention ranges from **6% to 18% in increments of 0.3 percentage points** (41 splits). The FFN receives the remaining budget. After debiting any removed MHA modules against the attention budget, we allocate the remaining compression in order of increasing ADS. We select the feasible policy with the lowest sum of ADS over the chosen layer-wise levels, removals, and compositions. ## W2. Does lower ADS predict lower held-out loss for both individual operations and complete policies? - **Test.** We computed ADS on the calibration data and loss on separate text. We then measured Spearman’s rank correlation: a positive value means that candidates with lower ADS tend to have lower loss, or a smaller increase in loss. **Table GPsy-1. Spearman rank correlation between ADS and held-out loss on LLaMA2-7B.** | Candidate set | Count | Quantities compared | Spearman ρ | | --------------------- | ---: | ------------------------- | ---------: | | All individual operations | 320 | ADS vs. increase in held-out loss | 0.86 | | MHA removal locations | 32 | Removal ADS vs. increase in held-out loss | 0.68 | | Complete policies, fixed attention/FFN split | 16 | Total ADS vs. held-out loss | 0.635 | - **Individual operations.** In each of 32 layers, we applied three retention levels (25.5%, 50%, 74.5%) to each of QK, VO, and FFN, and tested MHA removal separately. Across the resulting **320 operations**, ADS correlated with the increase in held-out loss (**ρ = 0.86**). Restricting the comparison to MHA removal at 32 different locations still yielded **ρ = 0.68**. - **Complete policies at a common budget split.** We fixed the attention/FFN split and varied the removal locations across 16 policies with closely matched parameter counts (4.7204–4.7222B). Total ADS correlated with final held-out loss (**ρ = 0.635**). Thus lower ADS predicted a smaller loss increase for single operations and tended to rank lower-loss complete policies ahead of higher-loss ones when choosing *where* to remove attention.

Thank you for the clarifications and additional experiments, which largely address my original concerns. However, I still find the overall method rather complex, and several design choices need stronger justification. These include how complete pruning policies are constructed and searched using ADS, as well as the added value of MRFC discussed by Reviewer MfA1. While the results support some individual components, it remains unclear whether the full combination offers enough benefit to justify its complexity. I am therefore inclined to keep my current score.

Follow-up to Reviewer GPsy

Thank you for taking the time to read our responses and examine the additional experiments so carefully. We appreciate your distinction between showing that individual components are effective and showing that their combined benefits justify the added complexity. We briefly summarize the rationale and evidence for the two design choices you highlighted.

  • Why use ADS to compare complete policies? To avoid compressing and evaluating a separate model for every candidate policy, SAP ranks candidates by summing the ADS values of their constituent operations. We tested whether this aggregate score reflects quality differences between complete policies, beyond its ability to rank individual operations. Across 16 policies with the same attention/FFN budget split but different attention-removal locations, the Spearman correlation between total ADS and held-out loss was ρ = 0.635 (Table GPsy-1). This supports using total ADS to decide where to reduce depth at a given budget split.

  • What does including MRFC offer in practice? SAP also considers policies that remove no attention layers. The reason to select MRFC is that, after attention removal, it combines two FFNs that would otherwise execute sequentially, thereby reducing generation latency. On LLaMA2-7B at 30% sparsity, MRFC reduced the number of executed blocks from 32 to 30 at the same parameter count as Attention Drop. Its seven-task mean accuracy was 0.24 percentage points lower (59.99% vs. 60.23%), while mean latency for generating 256 tokens was 1.5% lower with a 128-token prompt and 3.8% lower with a 1,024-token prompt. In a separate control, with both MRFC and the compression allocation held fixed, replacing FFN channel removal with SACR improved mean accuracy by 2.24 percentage points. These comparisons distinguish the roles of the two operators: MRFC reduces execution depth and latency, while SACR improves quality at the same allocation.

We will revise the manuscript to make both the evidence for ADS-based policy selection and the measured quality–latency trade-off explicit, so readers can judge when including MRFC is worthwhile.

Best regards,
The Authors

Comments on the paper

1 comment

General Response: summary of clarifications and additional experiments

We thank all reviewers for their careful reading and constructive feedback. Below, we consolidate the clarifications and additional evidence presented in our responses. Table labels refer to the reviewer-specific rebuttals. Component ablations and timing results below use LLaMA2-7B at 30% sparsity; pp denotes percentage points.

Concern (reviewers)What we added or clarifiedKey finding
Novelty beyond existing merging methods (MfA1 W1)Credited prior merging and attention-removal work and clarified the roles of SACR, MRFC, and ADS/SAP (MfA1 W1–W2).The contribution is the specific reconstruction and allocation procedures: SACR transfers information from lower-importance channels, MRFC composes neighboring FFNs, and ADS/SAP selects compression locations and amounts.
Operator versus allocation contributions (MfA1 W5/Q3; ZzHR W1/Q1)Varied FFN operators at fixed allocation, and allocation with operators fixed (Tables MfA1-4–5; ZzHR-1, ZzHR-3).SACR improves accuracy by 2.00–2.24 pp over channel removal at fixed allocation. ADS/SAP separately adds 0.73–0.76 pp over uniform allocation at similar parameter budgets. Olica + ADS/SAP performs best in the allocation comparison, demonstrating that the allocation benefit transfers across operators.
MRFC versus simpler alternatives (MfA1 W2/Q2; ZzHR W1/Q1; GPsy follow-up)Added operator comparisons, ridge-fitting sample sweeps across five models, and generation timings (Table MfA1-1; Tables ZzHR-1–2).MRFC executes 30 instead of 32 blocks and lowers mean latency by 1.5–3.8% versus Attention Drop, with 0.24 pp lower accuracy at the default fit. Increasing C4 ridge-fitting samples from 128 to 8,192 narrows the LLaMA2-7B PPL-128 gap from 0.62 to 0.22, with depth unchanged.
Whether ADS predicts held-out quality (MfA1 W3; GPsy W2)Compared calibration ADS with loss on separate text for individual operations, removal locations, and complete policies (Tables MfA1-2; GPsy-1).Spearman correlations are 0.86 across 320 operations, 0.68 across 32 MHA-removal locations, and 0.635 across 16 complete policies with a fixed attention/FFN budget split. These results support ADS-based candidate ranking within the tested settings.
SACR and SAP implementation and complexity (GPsy W1 and follow-up)Specified channel importance, pivot selection, construction and application of the gate/up mixing matrices, ridge output reconstruction, and SAP candidate generation (GPsy W1).SAP ranks policies using precomputed operation-level ADS, avoiding construction and evaluation of a separate compressed model for every policy. The specified search includes zero-removal policies, nonadjacent removal subsets, and attention/FFN budget splits.
SVD versus channel selection in VO compression (MfA1 Q1 and follow-up)Corrected the earlier rebuttal wording by explaining the inspected LLaMA2 implementation’s SVD-derived basis (MfA1 follow-up, Q1).VO compression uses activation-weighted SVD within each head, with the basis absorbed into V. ADS/SAP determines how many directions to retain; calibration-based importance determines which directions. The selection operates in an SVD-derived basis.
GQA baseline fairness and unfolding (MfA1 W4/Q4)Tuned Olica using held-out loss and added a comparison retaining native GQA for both methods; clarified that this is separate from the original Table 6 experiments (Table MfA1-3; MfA1 follow-up, Q4).On Qwen3-8B, reevaluation and tuning yield 163.5 PPL-128 for Olica versus 75.93 for PRISM under the same additional evaluation conditions. Both methods reduce KV-cache elements relative to dense in this comparison.
Quality at longer evaluation lengths (MfA1 Q5)Added WikiText-2 PPL at length 2,048 (Tables MfA1-3, MfA1-7).In the native-GQA comparison, PRISM achieves 16.66 vs. 18.93 on LLaMA-3.1-8B and 34.82 vs. 67.7 on Qwen3-8B against tuned Olica. On LLaMA2-7B, the methods are nearly tied at 21% sparsity; PRISM is worse at 30% and better at 39%.
Timing protocol and stability (ZzHR W2/Q2)Specified one RTX 4090, FP16, batch size 1, three warm-ups, ten timed runs, 128/1,024-token inputs, and 256 generated tokens; repeated measurements across sessions (Table ZzHR-4).PRISM’s mean latency is 4.4–4.8% lower than Olica’s, with a lower median at both input lengths in all three sessions containing both models.

Together, these controls separate SACR’s reconstruction benefit, ADS/SAP’s allocation benefit, and MRFC’s measured depth–quality trade-off. We will incorporate these distinctions and the additional evidence into the revision. We thank the reviewers again and welcome any remaining questions.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.