- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper studies the cause of activation sparsity in LLMs does not translate into proportional end-to-end speedups. It introduces a fixed-cost term to characterize the overhead of conditional skipping, and builds an inference stack based on a lean row-skipping kernel, sparsity-aware int8 quantization, CUDA graphs, and a temporal FFN reuse mechanism with cross-step hidden-state similarity. The evaluation on ProSparse-LLaMA-2 reports up to 1.38× end-to-end speed over the official graph baseline and 1.55× over TEAL.
Critical strengths
None.
Critical weaknesses
-
The presentation is poor. All the figures are not properly sized and difficult to read. The paper is also not organized clearly. It's hard to comprehend the key insight of the proposed method. Moreover, the technical novelty is limited, as the final design largely builds on existing techniques such as CUDA graphs, quantization.
-
The end-to-end evaluation is not well controlled. In Sec. 4.3 / Table 1, the final L3 stack combines multiple optimizations, while the reported 1.38× over the official baseline and 1.55× over TEAL use bf16 baselines against the proposed int8 stack. It is therefore difficult to know how much of the gain comes from the newly proposed optimizations, and the accuracy impact of FFN reuse is not sufficiently evaluated under realistic free-running generation.
Questions for authors
See weaknesses.