acceptodds
Under review as a conference paper at ICLR 2027

PRISM: Principal-Residual Activation Sparsity via Structure-Aware Budget Allocation for Efficient LLM Inference

Abstract

Training-free activation sparsity accelerates Large Language Model (LLM) inference by skipping computations on selected activation channels, without requiring additional training. Existing methods usually construct sparse computation patterns by selecting channels in the original activation coordinates. However, under high sparsity, the few retained channels may fail to cover sufficient output-relevant information, leading to large projection errors. Beyond the issue of channel selection, existing sparsity allocation methods commonly estimate module sensitivity with a single metric, overlooking structural dependencies inside Transformer blocks and the impact of error propagation. To address these limitations, we propose PRISM, a training-free activation sparsity framework with principal-residual computation and structure-aware sparsity allocation. PRISM first preserves low-dimensional principal components of linear-layer inputs through a compact dense path, and then applies sparse computation only to the residual activation. This dual-path design reduces the information loss caused by direct channel selection. PRISM further allocates sparsity budgets according to structural dependencies in Attention and MLP modules, avoiding overly aggressive sparsification of sensitive components. Experiments on representative LLMs show that PRISM better preserves model performance under high sparsity while providing practical inference speedup. At 75% sparsity, PRISM improves average reasoning accuracy by 3.7 points over the strongest sparse baseline and achieves up to 1.82 end-to-end generation speedup.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Model Inference, Activation Sparsity, Efficient Inference, Sparse Computation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_gQMsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This study proposes PRISM, a training-free activation sparsity method for LLM inference. It combines PCA-based decomposition with Top-K residual sparsification and structure-aware sparsity allocation to improve inference efficiency. Experiments demonstrate superior accuracy over existing methods while achieving substantial end-to-end inference speedup.

Critical strengths

The principal-residual decomposition is simple, clearly motivated by the output-retention analysis, and adds little memory overhead. The evaluation covers six models from several families, and the paper includes useful diagnostics on calibration sensitivity, principal-rank selection, and runtime breakdown.

Critical weaknesses
  1. Sparsity levels are matched in FLOPs, but decoding is largely memory-bound, so equal FLOPs do not imply equal latency. The paper’s own breakdown shows that the principal path accounts for about 2% of the computation but 11.5% of the end-to-end time. Yet all reported speedups are measured against dense inference only, with no sparse baseline measured. Meanwhile, accuracy is reported only at 60% and 75% sparsity. The headline 3.7-point gain comes from 75%, where several models show severe degradation. At 60%, the gain significantly shrinks, and no accuracy results are reported at 40–50%, the range where activation sparsity is typically deployed. The paper therefore does not show that PRISM offers a better accuracy–latency trade-off than existing methods at a practically relevant operating point.

  2. The baseline comparison is insufficient, and the contribution of each component is unclear. Specifically, the baselines use a unified Top-K selection rule, which modifies the original routing of some methods, while the treatment of their original allocation procedures is not described. Also, recent relevant methods such as WAS (EMNLP 2025) and WINA (ICLR 2026) are not included in the comparison.

Questions for authors
  1. Can the authors report the end-to-end latency of TEAL, R-Sparse, and LaRoSA under the same hardware and decoding settings, using comparably optimized implementations, and compare accuracy at matched latency?

  2. Can the authors report accuracy at 40% and 50% sparsity, together with an accuracy–speedup curve against the baselines?

  3. Did the TEAL baseline use its original greedy allocation, and was R-Sparse’s low-rank path included in the compute budget? How do the reproduced baseline results compare with those reported in the original papers, particularly for LaRoSA?

  4. Can the authors separate the two phases of the structure-aware allocation (projection-level initialization vs. structural correction) in the accuracy ablation, and compare against TEAL’s greedy allocation applied on top of the dual path?

4 comments
AuthorsOpenReview ↗

Response to Reviewer gQMs

We thank the reviewer for recognizing PRISM's design motivation and evaluation coverage. Below, we address the questions and provide an additional comparison with a recent baseline.

1. End-to-End Latency

We benchmark PRISM, TEAL, and R-Sparse at 60% target sparsity in a common inference framework adapted from public implementations. These latency measurements use updated, optimized kernels. Hardware and decoding settings follow the paper: batch size 1, a 1,024-token dense prefill, and 100 generated tokens. Latency includes prefill, decoding, and online channel selection. Values are in seconds.

ModelDensePRISMTEALR-Sparse
LLaMA-2-7B1.6461.0861.0181.177
LLaMA-3-8B1.7721.1721.1471.289
Mistral-7B1.6931.1011.0221.216

PRISM's end-to-end generation throughput is approximately 85–92 tokens/s. It is faster than Dense and R-Sparse, but slightly slower than TEAL.

The public LaRoSA code we used lacks a complete optimized end-to-end benchmark. Our reproduction yields limited acceleration with rotation overhead included, so we omit its latency.

2. Accuracy at 40% and 50% Sparsity

Using the paper's evaluation settings, we report seven-task accuracy at 40% and 50% sparsity on LLaMA-2-7B and LLaMA-3-8B. Scores are percentages; Avg7 is their equally weighted mean.

LLaMA-2-7B

SparsityMethodARC-CARC-EHellaSwagPIQAWinoGrandeGSM8KMMLUAvg7
0%Dense46.4274.6276.0279.1168.9013.9545.8457.84
40%TEAL43.6972.7774.0477.7368.2711.3042.7055.79
40%R-Sparse44.3773.2373.6877.8668.2711.8443.5356.11
40%LaRoSA44.7173.4374.2577.9768.3811.3743.4156.22
40%PRISM44.6473.5874.6178.0168.3511.6943.5956.35
50%TEAL41.3069.2871.7477.1566.147.1338.6353.05
50%R-Sparse41.3070.8370.9077.1766.968.0440.7553.71
50%LaRoSA41.9471.0172.0977.2067.098.8740.5154.10
50%PRISM42.2371.7872.2677.3167.248.9541.0754.41

LLaMA-3-8B

SparsityMethodARC-CARC-EHellaSwagPIQAWinoGrandeGSM8KMMLUAvg7
0%Dense53.4177.6979.1980.7972.4550.0465.3068.41
40%TEAL49.3276.3676.4879.3871.8236.2461.4764.44
40%R-Sparse49.5776.3576.4678.8471.0335.3361.3264.13
40%LaRoSA50.0276.5476.7879.6071.2735.7961.5264.50
40%PRISM50.5177.1176.8979.7971.9736.7261.8564.98
50%TEAL47.8773.5773.4377.6969.4524.2355.4960.25
50%R-Sparse46.6574.2872.8876.9769.3024.4956.2760.12
50%LaRoSA48.7273.9174.2678.1869.1126.1656.4460.97
50%PRISM49.2175.2174.8378.5969.8728.6557.3161.95

PRISM achieves higher Avg7 than TEAL, R-Sparse, and LaRoSA on both models at both sparsity levels. LaRoSA's sparsity budget covers sparse projections and excludes rotation cost. Its rotations are still executed during inference.

We also report paired accuracy and speedup values. Speed tests use the paper's hardware environment and decoding settings. We adapt TEAL's and R-Sparse's public code and include online Top-K selection overhead. Each row below gives a point for the accuracy-speedup comparison. Speedups are relative to each model's dense reference.

ModelSparsityMethodAvg7 (%)Speedup (x)
LLaMA-2-7B0%Dense57.841.000
LLaMA-2-7B40%TEAL55.791.224
LLaMA-2-7B40%R-Sparse56.111.155
LLaMA-2-7B40%PRISM56.351.173
LLaMA-2-7B50%TEAL53.051.355
LLaMA-2-7B50%R-Sparse53.711.273
LLaMA-2-7B50%PRISM54.411.297
LLaMA-3-8B0%Dense68.411.000
LLaMA-3-8B40%TEAL64.441.229
LLaMA-3-8B40%R-Sparse64.131.157
LLaMA-3-8B40%PRISM64.981.174
LLaMA-3-8B50%TEAL60.251.362
LLaMA-3-8B50%R-Sparse60.121.288
LLaMA-3-8B50%PRISM61.951.311

PRISM is slightly slower than TEAL but achieves higher Avg7 at the same sparsity. Compared with R-Sparse, it achieves both higher Avg7 and higher speedup at these points. Its speedup grows with sparsity, while the paper's results show a larger accuracy advantage at 60% and 75%.

AuthorsOpenReview ↗

Response to Reviewer gQMs

3. Baseline Allocation, Compute Budget, and Reproduction

All methods use per-token Top-K selection, replacing calibrated fixed thresholds where applicable. This fixes the retained channel count and avoids input-dependent sparsity drift.

Our TEAL baseline retains the original block-wise greedy allocation. It uses greedy-generated nonuniform sparsity tables, rather than assigning uniform sparsity to all projections.

R-Sparse's linear-projection budget includes both sparse and low-rank branches. After deducting the sparse-branch cost, we select the compensation rank from the remaining budget. Both costs are recorded. The low-rank path is therefore not treated as free computation.

We compare shared-task averages (%) with the original papers below. Differences are our evaluation minus the original scores, in percentage points.

LaRoSA

We recompute the average over six shared tasks: ARC-C, ARC-E, HellaSwag, PIQA, WinoGrande, and MMLU.

ModelSparsityOriginal PaperOur EvaluationDifference
LLaMA-2-7B40%64.4563.69-0.76
LLaMA-2-7B50%63.0261.64-1.38
LLaMA-3-8B40%70.0169.29-0.72
LLaMA-3-8B50%67.8866.77-1.11

The shared-task averages are close, with gaps of 0.72-1.38 percentage points. We follow the paper and public implementation. Differences in allocation parameters and evaluation settings can affect the reproduced scores. Rotations remain part of inference, but their cost is excluded from the matched projection budget.

TEAL

We recompute our average over TEAL's six reported tasks: ARC-C, HellaSwag, PIQA, WinoGrande, GSM8K, and MMLU.

ModelSparsityOriginal PaperOur EvaluationDifference
LLaMA-2-7B40%55.4552.96-2.49
LLaMA-2-7B50%54.2650.35-3.91
LLaMA-3-8B40%66.2162.45-3.76
LLaMA-3-8B50%63.4258.03-5.39

Our averages are 2.49-5.39 points lower under different evaluation settings. TEAL uses 25-shot ARC-C and 10-shot HellaSwag, whereas we use zero-shot for both. Even before sparsification, dense LLaMA-2-7B scores 52.47 on ARC-C in TEAL's paper versus 46.42 here. Our per-token Top-K selection also differs from fixed-threshold routing. The score comparison therefore needs to account for both evaluation settings and channel selection.

R-Sparse

We follow the published method and public inference code. Across the same four settings, PIQA and WinoGrande scores differ from the original paper by at most 0.72 and 0.44 percentage points, respectively. Some other tasks show larger gaps, which need to be considered alongside differences in evaluation settings and search configurations. R-Sparse searches for layer-wise ranks and budget allocations, but its released configurations do not cover every tested model and sparsity. For missing settings, we construct configurations from the published method. The comparison therefore reflects these configuration differences as well as the shared evaluation settings used in our study.

4. Separating the Two Allocation Phases

We separate projection-level initialization from structural correction and apply TEAL's greedy allocation to the same dual-path framework. Within each model, all variants share the principal rank, dual-path implementation, compute budget, and evaluation settings. The budget includes principal-path cost. The greedy variant changes only the allocation procedure.

The table reports Avg7 (%) at 60% sparsity.

Allocation on the dual pathMistral-7BLLaMA-3-8B
Uniform allocation58.7452.61
Projection-level initialization59.9253.16
TEAL greedy allocation60.1253.95
Initialization + structural correction60.3254.74

Projection-level initialization improves over uniform allocation on both models. Structural correction adds 0.40-1.58 percentage points. The full method also exceeds TEAL's greedy allocation by 0.20-0.79 points on the same dual path.

The two phases are complementary: initialization allocates the budget using projection-level sensitivity, while structural correction accounts for dependencies between coupled modules.

Comparison with a Recent Baseline

WINA (ICLR 2026) is a training-free activation sparsity baseline. It combines activation magnitudes with weight-column norms and uses greedy allocation.

PRISM instead preserves an activation-derived principal component through a compact dense path and sparsifies the residual. Its structure-aware allocation also accounts for dependencies within attention and MLP modules.

On LLaMA-3-8B, PRISM achieves higher Avg7 (%) at both sparsity levels:

ModelSparsityPRISM Avg7WINA Avg7
LLaMA-3-8B60%54.7453.21
LLaMA-3-8B75%36.7234.15

I thank the authors for conducting additional experiments to address my questions. However, my main concerns raised in the Critical Weaknesses remain unresolved, particularly regarding the baseline comparisons. I briefly reviewed the revised manuscript, but it seems that these comparisons are still missing.

AuthorsOpenReview ↗

Response to Reviewer gQMs

Thank you for the follow-up. We respond below to the concerns you identified as unresolved in the Critical Weaknesses.

1. Compute Cost and End-to-End Latency

We agree that equal FLOPs do not imply equal latency. Compute matching controls arithmetic work, whereas end-to-end timing also captures memory access and execution overhead.

The paper's LLaMA-2-7B profile at 60% sparsity illustrates this distinction: the principal path requires approximately 1.98% of the original dense projection computation but accounts for 11.5% of PRISM's end-to-end time. At batch size 1, the small kernels in the principal path may be limited by parallel execution efficiency, memory access, and scheduling overhead. Their runtime therefore does not scale proportionally with FLOPs.

2. Accuracy at Matched Latency

We use TEAL at 60% sparsity as the latency reference on LLaMA-3-8B. We adjust PRISM's sparsity experimentally to 63%. Its end-to-end latency closely matches that of TEAL under the same hardware and decoding settings. The following table reports the full seven-task evaluation, using the paper's accuracy evaluation settings.

ModelMethodSparsityAvg7 (%)
LLaMA-3-8BTEAL60%51.34
LLaMA-3-8BPRISM63%52.27

PRISM's higher Avg7 at these closely matched latencies indicates a more favorable accuracy-latency trade-off in this setting.

For R-Sparse, our previously reported comparisons on LLaMA-2-7B and LLaMA-3-8B at 40% and 50% sparsity show that PRISM achieves both higher Avg7 and higher speedup. This additional experiment therefore focuses on matching TEAL, which is faster at the same sparsity.

For LaRoSA, the public code used in our reproduction lacks a complete optimized end-to-end benchmark. Our adapted implementation yields limited acceleration when rotation overhead is included. We have therefore not obtained a comparably optimized end-to-end implementation and omit LaRoSA from the latency comparison.

3. Allocation and Token-Wise Channel Selection

Our main reason for using per-token Top-K is to prevent input-dependent sparsity fluctuations under fixed thresholds. As shown in Appendix B (Figure 6), a fixed threshold can retain different numbers of channels across tokens, so the same nominal sparsity need not correspond to a consistent compute budget. Appendix G.7 describes our use of Top-K to address this issue.

For methods that originally use fixed thresholds, we retain the nonuniform budgets produced by their allocation procedures and use Top-K to enforce the prescribed channel counts during inference. The modification is confined to online channel selection; it does not replace the allocation stage or the subsequent sparse and compensation computations. Its purpose is to make realized sparsity follow the allocated budget more consistently, while preserving each method's allocation design.

  • TEAL: We retain the nonuniform tables generated by its original block-wise greedy allocation. Top-K follows each projection's assigned budget; it does not replace the greedy-generated allocation.
  • R-Sparse: We retain the configured sparse/low-rank budget split and determine the compensation rank from the remaining budget. Both paths are counted. Top-K is applied after this configuration is set; it does not replace budget allocation or low-rank compensation. Released configurations are used where available, and missing settings follow the published method.
  • LaRoSA: We retain its configured relative allocation ratios and map them to the target projection budget. Top-K executes the resulting channel budget in the rotated coordinates, without replacing the allocation procedure or removing rotation.
Reviewer_EU5sOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

PRISM is a training-free activation-sparsity framework for LLM inference. The central claim is that Top-K channel selection in raw activation coordinates breaks down at high sparsity, because the few retained channels miss output-relevant information and projection error grows sharply. The paper instead decomposes each target projection input into a low-rank PCA principal part, kept dense through a compact principal path, and a residual that is sparsified with Top-K. Offline calibration on a small unlabeled corpus learns per-site PCA bases and projected weights; a compute-constrained rule then selects the principal rank, and a structure-aware allocator distributes the residual budget across projection groups (QK/VO/GU/D) using per-projection sensitivity plus bounded structural corrections derived from group-level signals. Sparsity is reported as compute-equivalent global sparsity that counts both the principal and residual paths. Evaluation covers LLaMA-2-7B/13B, LLaMA-3-8B, Mistral-7B, Qwen2.5-7B and Gemma-2-9B at 60% and 75% sparsity on five commonsense tasks (0-shot), GSM8K and MMLU (5-shot), and WikiText2 perplexity, against dense and prior work including TEAL, R-Sparse and LaRoSA, supported by rank, calibration and allocation ablations. Triton kernels are adapted from TEAL with end-to-end speedups 1.46x at 60% and 1.78x at 75% on LLaMA-2-7B/RTX4090/bs1. Code is released with CPU unit tests.

Critical strengths
  1. The principal-residual reframing is a genuine idea, well motivated and carefully executed. The motivation study shows residual sparsification cuts linear-projection error relative to raw-coordinate Top-K, which gives the design a concrete empirical footing. The dual-path construction then preserves a dense low-rank floor, so accuracy degrades gracefully as sparsity rises into the 60-75% regime where magnitude-based pruning typically collapses. This matters because it opens a new axis of what to sparsify. The rank-selection rule is also honest about cost: it stays within an explicit compute budget, which keeps the efficiency claim grounded.
  2. The evaluation goes beyond MAC-counting and backs the claim with a real system artifact. Six models across four families, reasoning-heavy suites in GSM8K and MMLU alongside the five commonsense tasks, plus dedicated calibration-sensitivity, rank and allocation ablations. The measured end-to-end Triton throughput, reaching up to 1.82x at 75% sparsity with a detailed breakdown showing the principal path at only around 2% of dense compute, is accompanied by a candid discussion of its disproportionate time share at batch 1 and the added memory cost. The released calibration-to-eval chain with pinned configs makes the whole pipeline auditable, which is a meaningful plus for an efficiency paper where reproduction is often the bottleneck.
Critical weaknesses
  1. Calibration transfer and offline search cost are under-characterized, which cuts at the training-free claim. The PCA bases, per-model ranks, and bound vectors are all fit on Alpaca calibration data with full-model forwards and full-vocabulary KL search over two budgets, yet the only cross-corpus evidence is a single LLaMA-2-7B setting where switching to WikiText-2 costs about 1.3 points. That is a useful datapoint but not a transfer argument: if the selected ranks or bounds move with calibration distribution, sequence length, or domain, the reported gains may not carry to new deployments. The paper also does not report wall-clock calibration cost in tokens or GPU-hours, even though the offline search runs full forwards with eager attention. Without cross-calibration deltas on at least one more model and an honest cost figure, it is hard to judge whether PRISM is a cheap drop-in recipe or a per-deployment tuning exercise.
  2. The speedup evidence is narrow and partly non-comparable, so the efficiency story overreaches as written. The detailed breakdown is batch-1 decode on a single GPU and model, a regime the authors themselves acknowledge is utilization-poor for the narrow principal path, and there is no evidence yet for larger batches, longer contexts, or prefill-heavy serving where sparsity trade-offs look quite different. On comparability, the LaRoSA numbers exclude its online rotation overhead while PRISM's own principal-path overhead is counted in the measured throughput, and the prefill policy is not stated per table even though the README in the released code warns that dense-prefill likelihood scores must not be read as sparse evaluations. Either broaden the serving envelope with at least one additional operating point or scope the headline claim to what the current numbers support.
Questions for authors
  1. Calibration transfer: beyond the single LLaMA-2-7B setting in Table 3, how do the selected ranks and bounds and the resulting downstream accuracy and perplexity change when calibration switches from Alpaca to WikiText-2 or C4, and when calibration sequence length changes? Please report the selected configurations and test deltas on at least one additional model.
  2. Compute-equivalent fairness: please report LaRoSA with its rotation cost both included and excluded at matched budgets, and give realized compute alongside target sparsity for each method on at least one model, to show what the headline sparsity numbers cost in practice.
  3. Prefill policy: for every main results table, please state the prefill mode and sparse ratio per method, and confirm that likelihood tasks such as MMLU and perplexity use identical prefill across methods.
  4. R-Sparse relationship: R-Sparse is already a baseline and is the closest prior dual sparse plus low-rank design, combining sparse channels with an SVD low-rank term and per-layer search. How does PRISM's PCA-on-activation plus residual Top-K differ from it in error behavior and budget allocation, and where does the empirical gap between the two come from?
  5. Offline pipeline cost: how many calibration tokens and GPU-hours do rank selection plus bound search take per model, and how sensitive are the main tables to halving that budget?
  6. Wider serving: do you have throughput at larger batch sizes such as 8 or 32, under KV-cache-heavy long contexts, to show the speedup holds outside batch-1 decode?
2 comments
AuthorsOpenReview ↗

Response to Reviewer EU5s

We thank the reviewer for recognizing PRISM's principal-residual design, compute accounting, and experimental evaluation. We address the reviewer's questions below.

1. Calibration Transfer

We add calibration-transfer experiments on Mistral-7B at 60% target sparsity. We replace Alpaca with WikiText-2 and separately shorten the maximum calibration sequence length from 512 to 128 tokens. WikiText-2 uses the same 512-token limit as the Alpaca reference. All conditions use the same rank and bound search procedure, total effective calibration-token budget, and downstream evaluation settings. PPL is evaluated on the same test data.

Mistral-7B

Calibration settingRankBounds (QK/VO/GU/D)Avg7 (%)PPL
Alpaca, reference240.035 / 0.035 / 0.035 / 0.03560.326.387
WikiText-2240.030 / 0.035 / 0.040 / 0.03559.896.485
Alpaca, shorter sequences240.040 / 0.040 / 0.035 / 0.04060.016.439

Switching to WikiText-2 and shortening the sequences reduce Avg7 by 0.43 and 0.31 percentage points, respectively. PPL increases by 0.098 and 0.052. The selected rank is relatively stable, while the group bounds show some variation. These results show stable downstream performance under the tested changes in calibration corpus and sequence length.

2. Compute-Equivalent Comparison

PRISM's principal path and R-Sparse's low-rank compensation are included in their compute budgets. The original LaRoSA comparison excludes rotation from the target budget, resulting in higher total computation at the same nominal sparsity. We additionally evaluate LaRoSA with rotation charged to the budget. Both settings execute rotation during inference.

For LLaMA-3-8B at 60% target sparsity, the table reports realized compute relative to the corresponding dense linear projections. These totals include the auxiliary paths, rather than only the sparse projections.

Method / Budget treatmentTarget sparsityCompute (% of dense)
TEAL60%40.004
R-Sparse, compensation included60%40.014
PRISM, principal path included60%39.997
LaRoSA, rotation excluded from budget60%55.389
LaRoSA, rotation included in budget60%40.006

LaRoSA's rotation adds approximately 15.385% of dense computation. Charging it to the fixed budget reduces the sparse-projection allowance from approximately 40% to 24.62%. The corresponding accuracy results are:

LaRoSA budget treatmentAvg7 (%)
Rotation excluded from budget52.77
Rotation included in budget39.03

Including rotation cost leaves less budget for LaRoSA's sparse projections and reduces accuracy. This distinguishes the nominal sparsity setting from the total computation required by the method.

3. Prefill Settings

Across the main accuracy tables, PRISM, TEAL, R-Sparse, and LaRoSA use the same task-specific prefill settings. ARC-C, ARC-E, HellaSwag, PIQA, WinoGrande, MMLU, and perplexity use Prefill99: approximately the first 1% of tokens are processed densely and the remaining 99% sparsely. GSM8K uses dense prefill and sparse autoregressive decoding.

The sparse portions use the target sparsity configuration specified in each table. The 99% ratio describes token coverage, not channel sparsity. MMLU and perplexity therefore use identical prefill settings across methods.

AuthorsOpenReview ↗

Response to Reviewer EU5s

4. Relationship to R-Sparse

Approximation error. R-Sparse uses low-rank weights to approximate discarded original-coordinate contributions. PRISM's principal path uses Wsmall=WPW_{\mathrm{small}}=WP to preserve the output contribution of the input activations' principal components. Only the residual is sparsified. This helps retain output-relevant information at high sparsity.

Budget allocation. Both methods count low-rank computation. R-Sparse searches the per-layer sparse/low-rank budget split using calibration perplexity. PRISM selects rank under the compute constraint, then allocates the remaining budget through projection-sensitivity initialization and bounded attention/MLP structural corrections.

Accuracy. On LLaMA-2-7B and LLaMA-3-8B, PRISM's Avg7 gains over R-Sparse increase from 1.73 and 3.61 percentage points at 60% sparsity to 2.85 and 5.08 points at 75%. This is consistent with stronger quality retention at high sparsity.

Computational reuse. Within q/k/v and gate/up groups sharing an input and PCA basis, PRISM reuses principal representations and residual construction. R-Sparse's independent weight bases generally do not support this direct reuse. Under the same hardware and decoding settings, the 75%-sparsity speedups of PRISM and R-Sparse are 1.783x versus 1.652x on LLaMA-2-7B, and 1.770x versus 1.708x on LLaMA-3-8B.

5. Offline Cost and Budget Sensitivity

We separately measure rank selection and projection-level initialization plus bound search. The table reports their one-time costs in GPU-hours, covering both 60% and 75% sparsity. It excludes PCA fitting and downstream evaluation. Selected ranks and sparsity tables remain fixed during inference, so this search is not repeated for each request.

ModelValid calibration tokensRank selectionInitialization + bound search
LLaMA-2-7B16,2680.1460.629
LLaMA-3-8B13,1350.1450.935
Mistral-7B15,5020.1480.818

To assess budget sensitivity, we halve the calibration-token budget and repeat calibration and configuration selection. Candidate sets, search rules, and downstream evaluation settings remain unchanged.

The following table compares full- and half-token budgets at 60% sparsity.

ModelFull-budget Avg7 (%)Half-budget Avg7 (%)Change (pp)
LLaMA-2-7B50.0349.37-0.66
LLaMA-3-8B54.7453.91-0.83
Mistral-7B60.3259.58-0.74

Halving the calibration-token budget reduces Avg7 by only 0.66-0.83 percentage points across the three models. PRISM thus maintains stable downstream performance with less calibration data.

6. Larger Batches and Long Contexts

Following the paper's speedup settings, we evaluate Mistral-7B at 60% sparsity with batch sizes 4 and 8. Speedups are relative to dense inference at the same batch size.

Batch sizePRISM speedup
41.149x
81.079x

PRISM still accelerates inference at batch sizes 4 and 8, although the gains are modest and decrease as batch size increases.

Larger batches improve weight reuse in dense computation, while different sequences' sparse channel selections limit batch-wide savings in weight access. TEAL also mentions limited scalability to larger batches. PRISM's kernels mainly target low-batch, latency-sensitive decoding. Operator fusion, data layouts, and scheduling remain areas for larger-batch optimization. The measured gains therefore reflect both batching characteristics and implementation efficiency.

Long contexts increase attention and KV-cache costs. PRISM primarily reduces weight access and computation in linear projections; it does not directly compress the KV cache. Overall speedup therefore depends on the runtime share of these projections. PRISM's projection optimizations are complementary to attention and KV-cache optimizations. Combining these with batch-oriented sparse kernels is a direction for extending long-context benefits.

Reviewer_sKExOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes PRISM, a training-free activation sparsity framework for efficient LLM inference. PRISM preserves a low-dimensional PCA-derived activation subspace through a compact dense path and applies Top-K sparsification to the residual. It further selects the principal rank under a compute budget and allocates residual sparsity using structure-aware sensitivity signals within attention and MLP modules. Experiments show improved average accuracy over existing baselines at 60% and 75% sparsity. It also reports up to 1.82× generation speedup over dense inference.

Critical strengths

PRISM offers a simple and intuitive principal–residual decomposition. It preserves principal components through a compact dense path and sparsifies only the residual, addressing information loss from direct channel selection. This provides a meaningful approach to improving sparse approximation without retraining.

Critical weaknesses
  1. Although PRISM outperforms the baselines at high sparsity levels, it still exhibits substantial performance degradation relative to the dense model (average accuracy drops by 20.60-31.69 at 75% sparsity). It does not address this degradation, leaving it unclear whether operating at such high sparsity levels is justified.

  2. The paper lacks a detailed analysis and discussion of the throughput-accuracy trade-off, making it difficult to assess whether the speedups justify the substantial accuracy losses, particularly at high sparsity.

Questions for authors
  1. Could the authors clarify why such high sparsity levels are necessary and valuable?

  2. How well does PRISM preserve information before substantial performance degradation occurs? Could the authors provide accuracy comparisons at lower sparsity levels, such as 25% and 40%?

  3. Could the authors provide a detailed analysis of the throughput-accuracy trade-off and compare models of different parameter sizes?

2 comments
AuthorsOpenReview ↗

Response to Reviewer sKEx

We sincerely thank the reviewer for recognizing the simplicity and intuitiveness of PRISM's principal-residual decomposition. We respond to the reviewer's questions below.

1. Motivation for High-Sparsity Evaluation

We include 75% sparsity as a demanding test of PRISM's ability to preserve model quality. Discarding most activation channels makes it harder to retain output-relevant information. This setting highlights the value of preserving principal components before sparsifying the residual. The paper's main experiments show higher average reasoning accuracy for PRISM than for the sparse baselines at 75% sparsity. This demonstrates stronger quality preservation under demanding conditions and motivates our high-sparsity evaluation.

2. Accuracy Preservation at Lower Sparsity

We add accuracy comparisons at 25% and 40% sparsity on LLaMA-2-7B, LLaMA-3-8B, and LLaMA-2-13B. The comparisons include TEAL, R-Sparse, LaRoSA, and PRISM, with Dense as the reference. We report all seven task scores and their equally weighted average (Avg7). All scores are in percent.

We use the same evaluation settings as in the paper. LaRoSA's reported sparsity is based on the sparse projection budget and excludes rotation cost. Rotation operations are still executed during inference.

LLaMA-2-7B

SparsityMethodARC-CARC-EHellaSwagPIQAWinoGrandeGSM8KMMLUAvg7
0%Dense46.4274.6276.0279.1168.9013.9545.8457.84
25%TEAL45.4874.0375.4678.1868.4312.6645.2657.07
25%R-Sparse45.5673.9975.4078.2469.1413.4745.4857.33
25%LaRoSA46.5073.9975.6178.7368.9013.4145.3757.50
25%PRISM45.9274.2875.8078.8968.8513.4345.5657.53
40%TEAL43.6972.7774.0477.7368.2711.3042.7055.79
40%R-Sparse44.3773.2373.6877.8668.2711.8443.5356.11
40%LaRoSA44.7173.4374.2577.9768.3811.3743.4156.22
40%PRISM44.6473.5874.6178.0168.3511.6943.5956.35

LLaMA-3-8B

SparsityMethodARC-CARC-EHellaSwagPIQAWinoGrandeGSM8KMMLUAvg7
0%Dense53.4177.6979.1980.7972.4550.0465.3068.41
25%TEAL52.6577.4478.7579.8772.1947.2964.4967.53
25%R-Sparse52.7977.1078.7379.7171.8545.7263.8867.11
25%LaRoSA53.0777.0678.7380.2272.1746.7064.6567.51
25%PRISM53.2877.5678.7980.3072.3347.7964.7867.83
40%TEAL49.3276.3676.4879.3871.8236.2461.4764.44
40%R-Sparse49.5776.3576.4678.8471.0335.3361.3264.13
40%LaRoSA50.0276.5476.7879.6071.2735.7961.5264.50
40%PRISM50.5177.1176.8979.7971.9736.7261.8564.98

LLaMA-2-13B

SparsityMethodARC-CARC-EHellaSwagPIQAWinoGrandeGSM8KMMLUAvg7
0%Dense49.1577.4879.3880.5872.1423.0555.1762.42
25%TEAL48.8277.1579.2580.3971.9022.5854.9262.14
25%R-Sparse48.8277.2678.9080.5272.0522.7254.7362.14
25%LaRoSA48.7177.3179.1280.4172.0822.8454.9462.20
25%PRISM48.8977.3579.2780.5072.0622.9054.9862.28
40%TEAL47.8376.3178.8679.8770.9021.2353.3361.19
40%R-Sparse47.7275.9277.4579.9270.8820.7653.1060.82
40%LaRoSA47.6576.0478.3579.7670.6921.4653.4261.05
40%PRISM48.0576.8978.9980.0971.0921.8553.8361.54

Across the three models, PRISM's Avg7 drops by less than 0.6 percentage points at 25% sparsity relative to Dense. At 40% sparsity, the drop remains within 3.5 percentage points. These results show good preservation of downstream performance at lower sparsity.

PRISM achieves higher Avg7 than TEAL, R-Sparse, and LaRoSA on all three models at both 25% and 40% sparsity. The paper's high-sparsity results show that this advantage grows as sparsity increases. It is more pronounced at 60% and 75%.

AuthorsOpenReview ↗

Response to Reviewer sKEx

3. Throughput-Accuracy Trade-Off Across Model Sizes

We adapted TEAL's and R-Sparse's public code for throughput evaluation. The hardware environment and decoding settings match those in the paper. We use dynamic Top-K to select channels for each token. Timing includes the online selection overhead, which moderates the speedup. The table reports speedup relative to each model's dense reference.

ModelMethodDense25%40%50%60%75%
LLaMA-2-7BPRISM1.0001.1091.1731.2971.4571.783
LLaMA-2-7BTEAL1.0001.1541.2241.3551.5371.875
LLaMA-2-7BR-Sparse1.0001.0921.1551.2731.3921.652
LLaMA-3-8BPRISM1.0001.1131.1741.3111.4541.770
LLaMA-3-8BTEAL1.0001.1641.2291.3621.5331.858
LLaMA-3-8BR-Sparse1.0001.1021.1571.2881.4321.708
LLaMA-2-13BPRISM1.0001.1171.1831.3191.4791.816
LLaMA-2-13BTEAL1.0001.1571.2281.3771.5621.860
LLaMA-2-13BR-Sparse1.0001.1021.1641.2951.4331.734

At 25% sparsity, PRISM achieves about 1.11x speedup. Avg7 drops by less than one percentage point across all three models. Speedup increases with sparsity, reaching up to 1.82x at 75%. PRISM is slightly slower than TEAL but maintains higher Avg7. Its accuracy advantage becomes more pronounced at higher sparsity. At the low-sparsity points shown, PRISM also outperforms R-Sparse in both speedup and Avg7. Lower sparsity favors accuracy preservation, while higher sparsity trades more accuracy for acceleration.

These benefits also extend to LLaMA-2-13B. At 40% sparsity, PRISM achieves about 1.18x speedup with an Avg7 drop of only 0.88 percentage points relative to Dense. Its speedup reaches about 1.82x at 75% sparsity. Together with the 7B and 8B results, this supports PRISM's ability to preserve quality while accelerating inference across model sizes.

The public LaRoSA code used in our reproduction lacks a complete optimized pipeline for end-to-end throughput evaluation. With rotation overhead included, our reproduction yields limited acceleration. We therefore omit LaRoSA from the throughput comparison.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.