- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes PRISM, a structured compression framework that combines depth reduction and channel compression without gradient-based retraining. It removes selected attention modules, composes adjacent FFNs, and reconstructs compressed channels to transfer information into retained weights. An activation-distortion score guides where and how much to compress. Experiments on several LLM families show competitive perplexity and downstream accuracy, including extensions to GQA models and models up to 32B. Measurements on LLaMA2-7B further report generation latency, memory use, and the compression pipeline's wall-clock cost.
Critical strengths
- Useful compression pipeline with broader model validation. PRISM combines inter-layer composition and intra-layer reconstruction, and Appendix F extends the main experiments to LLaMA-3.1 and Qwen3, including 32B. On Qwen3-32B, it reaches 66.10 average accuracy versus Olica's 64.53 at approximately 30% sparsity. This supports applicability beyond the older 7B models in the main table and shows that the benefit can extend to a substantially larger GQA model.
- Measured quality–efficiency trade-offs. Tables 2–3 report actual generation latency and compression-stage timing rather than relying only on parameter reduction. At 39% sparsity, PRISM improves Olica's PPL from 28.14 to 24.98 and latency from 5.13 to 4.90 seconds on the measured model. This provides direct evidence of a better operating point in that workload, and the stage breakdown identifies activation-distortion measurement as the main offline cost.
Critical weaknesses
- End-to-end component contributions are not isolated. Table 4 ablates the allocation metric, while supplementary Figure 6 evaluates MRFC/SACR using local activation distortion. These do not show how much each merging module contributes to final PPL, downstream accuracy, or latency at a matched compression budget. Because ADS is also the selection criterion, improving local ADS alone does not establish that both modules are necessary for the end-task gain. Representative matched-budget module ablations would strengthen the central combination claim.
- The practical efficiency advantage is difficult to assess or reproduce. Detailed timings cover only LLaMA2-7B generating 256 tokens; at 30% sparsity the Olica–PRISM latency difference is 5.71 versus 5.62 seconds. The manuscript does not specify the timing GPU, precision, batch size, input length, or repeated-run variability. These details are particularly important for a difference of about 1.6%, and their absence leaves both its stability and the GQA/larger-model efficiency benefit unverified.
Questions for authors
- Are existing matched-budget results available for MRFC-only, SACR-only, and their combination, using final task metrics rather than local ADS? If not, please qualify the component-level conclusions.
- Please specify the complete timing protocol for Tables 2–3/8, including hardware, precision, batch/input sizes, warm-up, and repeats. Is the 0.09-second advantage at 30% sparsity stable across runs?
5 comments
Response to Reviewer ZzHR (2/3): How do calibration, block count, and allocation affect the results?
W1/Q1 (continued). Does fitting MRFC on more samples narrow the quality gap to Attention Drop, and what do we gain from reducing depth?
Table ZzHR-2(c). MRFC PPL-128 / PPL-2048 as a function of ridge-fitting sample count on the four additional models. denotes fitting samples; a dash denotes an unreported result.
| Model | Ridge data | N=128 | N=512 | N=2,048 | N=8,192 |
|---|---|---|---|---|---|
| LLaMA-3.1-8B | BookCorpus | 40.03 / 16.86 | 36.46 / 15.70 | 36.30 / 15.61 | 36.24 / 15.58 |
| LLaMA-3.1-8B | C4 | 39.62 / 16.68 | 36.24 / 15.54 | 36.03 / 15.48 | 35.97 / 15.46 |
| Qwen3-8B | BookCorpus | 79.75 / 35.90 | 73.45 / 34.20 | 73.01 / 33.98 | 72.70 / 33.87 |
| Qwen3-8B | C4 | 78.63 / 35.04 | 72.55 / 33.48 | 72.15 / 33.35 | 72.10 / 33.35 |
| LLaMA2-13B | BookCorpus | 14.94 / 6.64 | 14.80 / 6.58 | 14.75 / 6.57 | — |
| LLaMA2-13B | C4 | 14.98 / 6.65 | 14.77 / 6.57 | 14.75 / 6.57 | — |
| Qwen3-32B | BookCorpus | 30.33 / 12.59 | 29.51 / 12.34 | 29.24 / 12.25 | — |
| Qwen3-32B | C4 | 29.88 / 12.46 | 29.39 / 12.29 | 29.12 / 12.21 | — |
- Effect of more fitting data. Increasing either BookCorpus or C4 from 128 to 2,048 samples lowered both PPLs on all four models. On Qwen3-8B, 512 C4 samples gave 72.55 / 33.48, below both Attention Drop (75.00 / 34.36) and unaligned concatenation (74.93 / 34.54). On LLaMA-3.1-8B, 8,192 C4 samples put MRFC below concatenation but above Attention Drop. On LLaMA2-13B and Qwen3-32B, the remaining gaps to Attention Drop at 2,048 C4 samples were 0.05 / 0.03 and 0.24 / 0.10, respectively.
Table ZzHR-2(d). Time to generate 256 tokens with each inter-layer operator on LLaMA2-7B at 30% sparsity. We used Hugging Face’s eager generate on one RTX 4090 (FP16, batch size 1). Values are means ± standard deviations across five sessions.
| Inter-layer operator | Blocks | 128-token prompt (s) | 1,024-token prompt (s) |
|---|---|---|---|
| MRFC | 30 | 6.435 ± 0.111 | 6.434 ± 0.193 |
| Attention Drop | 32 | 6.533 ± 0.074 | 6.689 ± 0.107 |
| Unaligned concat | 30 | 6.322 ± 0.270 | 6.449 ± 0.175 |
-
Fewer blocks, lower generation latency. Attention Drop leaves the two neighboring FFNs in sequence. MRFC composes them, reducing the number of blocks executed from 32 to 30. MRFC took 0.098 s less (1.5%) than Attention Drop on average with a 128-token prompt and 0.255 s less (3.8%) with a 1,024-token prompt. Unaligned concat, which also executes 30 blocks, was faster than Attention Drop at both prompt lengths.
-
Component result versus full-model result. On LLaMA2-7B, MRFC trades two blocks for a higher default PPL than Attention Drop; increasing the ridge sample count narrows that gap without adding the blocks back (Table ZzHR-2(a)). In the separate comparison of complete methods in manuscript Table 2, PRISM reached 18.16 PPL-128 at 30% sparsity, compared with 18.39 for Olica and 30.70 for CoMe. Those manuscript values do not use the best result from the ridge sweep.
Table ZzHR-3. Changing allocation while keeping the operators fixed. PPL and accuracy use the same aggregation as Table ZzHR-1. Within each operator family, the parameter counts differ by approximately 0.10–0.48% between allocation strategies.
| Operators | Allocation | Parameters (B) | PPL-128 | PPL-2048 | Accuracy (%) |
|---|---|---|---|---|---|
| Olica | Uniform | 4.7250 | 18.49 | 8.12 | 59.37 |
| Olica | ADS/SAP | 4.7298 | 18.07 | 7.97 | 60.10 |
| PRISM | Uniform | 4.7168 | 19.29 | 8.55 | 59.23 |
| PRISM | ADS/SAP | 4.7396 | 18.17 | 8.27 | 59.99 |
- Allocation has a separate effect. Switching from uniform allocation to ADS/SAP lowered PPL at both lengths for both operator families. Accuracy rose by 0.73 percentage points for Olica and by 0.76 for PRISM with MRFC and SACR. Olica + ADS/SAP performed best in this comparison. In the revision, we will distinguish this allocation effect from SACR’s gain over channel removal and from MRFC’s depth–quality trade-off.
Response to Reviewer ZzHR (3/3): Does PRISM’s latency advantage over Olica persist across runs?
W2/Q2. How was latency measured, and does the original 0.09-second advantage persist across runs?
- Original measurement. Manuscript Tables 2 and 8 report inference time including prompt processing and generation of 256 tokens. At 30% sparsity, Olica took 5.71 s and PRISM took 5.62 s, averaged over 10 runs on one RTX 4090, LLaMA2-7B, FP16, batch size 1, with a 128-token input. We retained the means but not the individual timings, so we cannot recover the variance of that original difference. Manuscript Table 3 instead reports the offline, per-stage cost of compressing one model.
- Repeat protocol. On the same device and at the same precision, we used Hugging Face’s eager
generatewith 128- and 1,024-token inputs and a fixed 256-token generation length. We randomized model order within each session and ran three warm-ups followed by ten timed runs per model. The speedup reference in Table ZzHR-4 is the effectively unpruned Olica implementation, which uses the same execution path as the compressed models.
Table ZzHR-4. Generation latency and speedup relative to the Olica Dense reference at two input lengths.
| Model | 128-token input: latency (s) | Speedup vs. reference | 1,024-token input: latency (s) | Speedup vs. reference |
|---|---|---|---|---|
| Olica Dense, nominal SR 0% | 6.683 ± 0.124 | 1.000× | 6.824 ± 0.128 | 1.000× |
| Olica, SR 30% | 6.469 ± 0.099 | 1.033 ± 0.025× | 6.583 ± 0.047 | 1.037 ± 0.021× |
| PRISM, SR 30% | 6.185 ± 0.212 | 1.080 ± 0.042× | 6.270 ± 0.051 | 1.088 ± 0.022× |
- Direct comparison with Olica at 30% sparsity. Using the mean latencies in the table, PRISM took 0.284 s less (4.39%) at 128 input tokens and 0.313 s less (4.75%) at 1,024. In all three sessions that included both compressed models, PRISM also had the lower median latency at both input lengths.
- What was reproducible? For each session, we divided Olica’s median over ten runs by PRISM’s median. The mean and between-session standard deviation of that ratio were 1.051 ± 0.015× at 128 input tokens and 1.060 ± 0.011× at 1,024. These measurements reproduce the direction of the advantage over Olica under this workload; they do not establish that the original difference is always exactly 0.09 s.
Response to Reviewer ZzHR (1/3): What do SACR and MRFC contribute at matched sparsity?
Thank you for asking for end-to-end evidence of each component’s contribution and for a measurement protocol that puts the reported 0.09-second latency difference in context. We address the two points separately below.
W1/Q1. At a matched compression budget, what do MRFC and SACR each contribute to final model quality?
Table ZzHR-1. Operator ablations at a matched budget. PPL is averaged over three calibration sets. Accuracy is the seven-task mean for the model compressed with the manuscript calibration set; all rows have 4.7396B parameters on that set.
Table ZzHR-2(a). LLaMA2-7B: varying only the ridge-fitting sample count. Unlike the three-set means in Table ZzHR-1, this experiment fixes the removal locations, allocation, and the calibration data for all other steps (128 BookCorpus samples). The MRFC regressions use dense-model inputs.
Table ZzHR-2(b). Inter-layer operators on four additional models at 30% sparsity. Each method uses the same calibration set within a model (seed 0). Values are WikiText-2 PPL at lengths 128 / 2,048. Each Δ is MRFC PPL-128 minus the respective baseline PPL-128.