- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This study proposes PRISM, a training-free activation sparsity method for LLM inference. It combines PCA-based decomposition with Top-K residual sparsification and structure-aware sparsity allocation to improve inference efficiency. Experiments demonstrate superior accuracy over existing methods while achieving substantial end-to-end inference speedup.
Critical strengths
The principal-residual decomposition is simple, clearly motivated by the output-retention analysis, and adds little memory overhead. The evaluation covers six models from several families, and the paper includes useful diagnostics on calibration sensitivity, principal-rank selection, and runtime breakdown.
Critical weaknesses
-
Sparsity levels are matched in FLOPs, but decoding is largely memory-bound, so equal FLOPs do not imply equal latency. The paper’s own breakdown shows that the principal path accounts for about 2% of the computation but 11.5% of the end-to-end time. Yet all reported speedups are measured against dense inference only, with no sparse baseline measured. Meanwhile, accuracy is reported only at 60% and 75% sparsity. The headline 3.7-point gain comes from 75%, where several models show severe degradation. At 60%, the gain significantly shrinks, and no accuracy results are reported at 40–50%, the range where activation sparsity is typically deployed. The paper therefore does not show that PRISM offers a better accuracy–latency trade-off than existing methods at a practically relevant operating point.
-
The baseline comparison is insufficient, and the contribution of each component is unclear. Specifically, the baselines use a unified Top-K selection rule, which modifies the original routing of some methods, while the treatment of their original allocation procedures is not described. Also, recent relevant methods such as WAS (EMNLP 2025) and WINA (ICLR 2026) are not included in the comparison.
Questions for authors
-
Can the authors report the end-to-end latency of TEAL, R-Sparse, and LaRoSA under the same hardware and decoding settings, using comparably optimized implementations, and compare accuracy at matched latency?
-
Can the authors report accuracy at 40% and 50% sparsity, together with an accuracy–speedup curve against the baselines?
-
Did the TEAL baseline use its original greedy allocation, and was R-Sparse’s low-rank path included in the compute budget? How do the reproduced baseline results compare with those reported in the original papers, particularly for LaRoSA?
-
Can the authors separate the two phases of the structure-aware allocation (projection-level initialization vs. structural correction) in the accuracy ablation, and compare against TEAL’s greedy allocation applied on top of the dual path?
4 comments
Response to Reviewer gQMs
3. Baseline Allocation, Compute Budget, and Reproduction
All methods use per-token Top-K selection, replacing calibrated fixed thresholds where applicable. This fixes the retained channel count and avoids input-dependent sparsity drift.
Our TEAL baseline retains the original block-wise greedy allocation. It uses greedy-generated nonuniform sparsity tables, rather than assigning uniform sparsity to all projections.
R-Sparse's linear-projection budget includes both sparse and low-rank branches. After deducting the sparse-branch cost, we select the compensation rank from the remaining budget. Both costs are recorded. The low-rank path is therefore not treated as free computation.
We compare shared-task averages (%) with the original papers below. Differences are our evaluation minus the original scores, in percentage points.
LaRoSA
We recompute the average over six shared tasks: ARC-C, ARC-E, HellaSwag, PIQA, WinoGrande, and MMLU.
| Model | Sparsity | Original Paper | Our Evaluation | Difference |
|---|---|---|---|---|
| LLaMA-2-7B | 40% | 64.45 | 63.69 | -0.76 |
| LLaMA-2-7B | 50% | 63.02 | 61.64 | -1.38 |
| LLaMA-3-8B | 40% | 70.01 | 69.29 | -0.72 |
| LLaMA-3-8B | 50% | 67.88 | 66.77 | -1.11 |
The shared-task averages are close, with gaps of 0.72-1.38 percentage points. We follow the paper and public implementation. Differences in allocation parameters and evaluation settings can affect the reproduced scores. Rotations remain part of inference, but their cost is excluded from the matched projection budget.
TEAL
We recompute our average over TEAL's six reported tasks: ARC-C, HellaSwag, PIQA, WinoGrande, GSM8K, and MMLU.
| Model | Sparsity | Original Paper | Our Evaluation | Difference |
|---|---|---|---|---|
| LLaMA-2-7B | 40% | 55.45 | 52.96 | -2.49 |
| LLaMA-2-7B | 50% | 54.26 | 50.35 | -3.91 |
| LLaMA-3-8B | 40% | 66.21 | 62.45 | -3.76 |
| LLaMA-3-8B | 50% | 63.42 | 58.03 | -5.39 |
Our averages are 2.49-5.39 points lower under different evaluation settings. TEAL uses 25-shot ARC-C and 10-shot HellaSwag, whereas we use zero-shot for both. Even before sparsification, dense LLaMA-2-7B scores 52.47 on ARC-C in TEAL's paper versus 46.42 here. Our per-token Top-K selection also differs from fixed-threshold routing. The score comparison therefore needs to account for both evaluation settings and channel selection.
R-Sparse
We follow the published method and public inference code. Across the same four settings, PIQA and WinoGrande scores differ from the original paper by at most 0.72 and 0.44 percentage points, respectively. Some other tasks show larger gaps, which need to be considered alongside differences in evaluation settings and search configurations. R-Sparse searches for layer-wise ranks and budget allocations, but its released configurations do not cover every tested model and sparsity. For missing settings, we construct configurations from the published method. The comparison therefore reflects these configuration differences as well as the shared evaluation settings used in our study.
4. Separating the Two Allocation Phases
We separate projection-level initialization from structural correction and apply TEAL's greedy allocation to the same dual-path framework. Within each model, all variants share the principal rank, dual-path implementation, compute budget, and evaluation settings. The budget includes principal-path cost. The greedy variant changes only the allocation procedure.
The table reports Avg7 (%) at 60% sparsity.
| Allocation on the dual path | Mistral-7B | LLaMA-3-8B |
|---|---|---|
| Uniform allocation | 58.74 | 52.61 |
| Projection-level initialization | 59.92 | 53.16 |
| TEAL greedy allocation | 60.12 | 53.95 |
| Initialization + structural correction | 60.32 | 54.74 |
Projection-level initialization improves over uniform allocation on both models. Structural correction adds 0.40-1.58 percentage points. The full method also exceeds TEAL's greedy allocation by 0.20-0.79 points on the same dual path.
The two phases are complementary: initialization allocates the budget using projection-level sensitivity, while structural correction accounts for dependencies between coupled modules.
Comparison with a Recent Baseline
WINA (ICLR 2026) is a training-free activation sparsity baseline. It combines activation magnitudes with weight-column norms and uses greedy allocation.
PRISM instead preserves an activation-derived principal component through a compact dense path and sparsifies the residual. Its structure-aware allocation also accounts for dependencies within attention and MLP modules.
On LLaMA-3-8B, PRISM achieves higher Avg7 (%) at both sparsity levels:
| Model | Sparsity | PRISM Avg7 | WINA Avg7 |
|---|---|---|---|
| LLaMA-3-8B | 60% | 54.74 | 53.21 |
| LLaMA-3-8B | 75% | 36.72 | 34.15 |
I thank the authors for conducting additional experiments to address my questions. However, my main concerns raised in the Critical Weaknesses remain unresolved, particularly regarding the baseline comparisons. I briefly reviewed the revised manuscript, but it seems that these comparisons are still missing.
Response to Reviewer gQMs
Thank you for the follow-up. We respond below to the concerns you identified as unresolved in the Critical Weaknesses.
1. Compute Cost and End-to-End Latency
We agree that equal FLOPs do not imply equal latency. Compute matching controls arithmetic work, whereas end-to-end timing also captures memory access and execution overhead.
The paper's LLaMA-2-7B profile at 60% sparsity illustrates this distinction: the principal path requires approximately 1.98% of the original dense projection computation but accounts for 11.5% of PRISM's end-to-end time. At batch size 1, the small kernels in the principal path may be limited by parallel execution efficiency, memory access, and scheduling overhead. Their runtime therefore does not scale proportionally with FLOPs.
2. Accuracy at Matched Latency
We use TEAL at 60% sparsity as the latency reference on LLaMA-3-8B. We adjust PRISM's sparsity experimentally to 63%. Its end-to-end latency closely matches that of TEAL under the same hardware and decoding settings. The following table reports the full seven-task evaluation, using the paper's accuracy evaluation settings.
| Model | Method | Sparsity | Avg7 (%) |
|---|---|---|---|
| LLaMA-3-8B | TEAL | 60% | 51.34 |
| LLaMA-3-8B | PRISM | 63% | 52.27 |
PRISM's higher Avg7 at these closely matched latencies indicates a more favorable accuracy-latency trade-off in this setting.
For R-Sparse, our previously reported comparisons on LLaMA-2-7B and LLaMA-3-8B at 40% and 50% sparsity show that PRISM achieves both higher Avg7 and higher speedup. This additional experiment therefore focuses on matching TEAL, which is faster at the same sparsity.
For LaRoSA, the public code used in our reproduction lacks a complete optimized end-to-end benchmark. Our adapted implementation yields limited acceleration when rotation overhead is included. We have therefore not obtained a comparably optimized end-to-end implementation and omit LaRoSA from the latency comparison.
3. Allocation and Token-Wise Channel Selection
Our main reason for using per-token Top-K is to prevent input-dependent sparsity fluctuations under fixed thresholds. As shown in Appendix B (Figure 6), a fixed threshold can retain different numbers of channels across tokens, so the same nominal sparsity need not correspond to a consistent compute budget. Appendix G.7 describes our use of Top-K to address this issue.
For methods that originally use fixed thresholds, we retain the nonuniform budgets produced by their allocation procedures and use Top-K to enforce the prescribed channel counts during inference. The modification is confined to online channel selection; it does not replace the allocation stage or the subsequent sparse and compensation computations. Its purpose is to make realized sparsity follow the allocated budget more consistently, while preserving each method's allocation design.
- TEAL: We retain the nonuniform tables generated by its original block-wise greedy allocation. Top-K follows each projection's assigned budget; it does not replace the greedy-generated allocation.
- R-Sparse: We retain the configured sparse/low-rank budget split and determine the compensation rank from the remaining budget. Both paths are counted. Top-K is applied after this configuration is set; it does not replace budget allocation or low-rank compensation. Released configurations are used where available, and missing settings follow the published method.
- LaRoSA: We retain its configured relative allocation ratios and map them to the target projection budget. Top-K executes the resulting channel budget in the rotated coordinates, without replacing the allocation procedure or removing rotation.
Response to Reviewer gQMs
We thank the reviewer for recognizing PRISM's design motivation and evaluation coverage. Below, we address the questions and provide an additional comparison with a recent baseline.
1. End-to-End Latency
We benchmark PRISM, TEAL, and R-Sparse at 60% target sparsity in a common inference framework adapted from public implementations. These latency measurements use updated, optimized kernels. Hardware and decoding settings follow the paper: batch size 1, a 1,024-token dense prefill, and 100 generated tokens. Latency includes prefill, decoding, and online channel selection. Values are in seconds.
PRISM's end-to-end generation throughput is approximately 85–92 tokens/s. It is faster than Dense and R-Sparse, but slightly slower than TEAL.
The public LaRoSA code we used lacks a complete optimized end-to-end benchmark. Our reproduction yields limited acceleration with rotation overhead included, so we omit its latency.
2. Accuracy at 40% and 50% Sparsity
Using the paper's evaluation settings, we report seven-task accuracy at 40% and 50% sparsity on LLaMA-2-7B and LLaMA-3-8B. Scores are percentages; Avg7 is their equally weighted mean.
LLaMA-2-7B
LLaMA-3-8B
PRISM achieves higher Avg7 than TEAL, R-Sparse, and LaRoSA on both models at both sparsity levels. LaRoSA's sparsity budget covers sparse projections and excludes rotation cost. Its rotations are still executed during inference.
We also report paired accuracy and speedup values. Speed tests use the paper's hardware environment and decoding settings. We adapt TEAL's and R-Sparse's public code and include online Top-K selection overhead. Each row below gives a point for the accuracy-speedup comparison. Speedups are relative to each model's dense reference.
PRISM is slightly slower than TEAL but achieves higher Avg7 at the same sparsity. Compared with R-Sparse, it achieves both higher Avg7 and higher speedup at these points. Its speedup grows with sparsity, while the paper's results show a larger accuracy advantage at 60% and 75%.