- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes CheeSe, a channel permutation method for improving the accuracy of 2:4 semi-structured KV cache pruning. The authors observe that conventional 2:4 pruning suffers from significant accuracy degradation because important key channels tend to be grouped together. By rearranging channels according to their importance, CheeSe reduces this degradation while preserving the hardware efficiency of 2:4 sparsity. The method also incorporates output-aware selection to further improve pruning quality.
Experiments across five language models show that CheeSe achieves accuracy comparable to unstructured pruning, while offering better effective KV cache compression and a 1.80× speedup in decode attention using an existing sparse kernel.
Critical strengths
-
Simple and practical approach. The proposed method is straightforward and does not require model retraining. The permutation can be computed offline and reused across different inputs, with relatively low runtime overhead.
-
Consistent experimental improvements. The method substantially reduces the accuracy degradation of conventional 2:4 pruning across five models. The results suggest that channel permutation is an effective way to make semi-structured KV cache pruning more practical.
-
Hardware compatibility. By preserving the 2:4 sparsity pattern, CheeSe can leverage existing GPU sparse computation support, providing benefits in both memory efficiency and attention computation.
Critical weaknesses
-
Limited novelty. Channel permutation has already been explored in previous work on semi-structured weight pruning. The main contribution of this paper is adapting an existing technique to the KV cache setting and demonstrating its effectiveness. While this application is useful, the algorithmic novelty is relatively limited.
-
Insufficient end-to-end efficiency evaluation. The paper reports a 1.80× speedup for decode attention and a relatively small overhead from permutation. However, it does not provide a comprehensive end-to-end inference evaluation. Since improving inference efficiency is a central motivation of the paper, it would be valuable to demonstrate the actual benefits in overall generation latency and throughput.
Questions for authors
-
Could the authors provide end-to-end inference latency or throughput measurements that account for the permutation and pruning overhead? It would be particularly helpful to evaluate different context lengths and batch sizes.
-
How much additional practical benefit does output-aware selection provide beyond channel permutation alone? The reported gains appear relatively small and are not statistically significant for individual models. Could the authors further clarify its importance in the overall method?
4 comments
Follow-Up to Your Review
Dear Reviewer 6QFZ,
We are sincerely grateful for your careful review. We have already posted our responses to each point you raised.
If your schedule permits, we would be truly thankful if you could let us know whether any concerns remain. We would be more than happy to provide further explanation.
With our sincere appreciation, The Authors of Submission 48
Thank you for the detailed response and the additional experiments.
The new end-to-end efficiency results address my main concern regarding the practical speedup of CheeSe. In particular, the 1.21× end-to-end speedup at 32K context length with batch size 8 provides stronger evidence of its practical value. I also appreciate the clarification of the role of OA selection and its benefits on specific tasks.
Although the algorithmic novelty remains relatively limited, I believe the simplicity of the method, its consistent accuracy improvements, and the additional efficiency results make a meaningful contribution.
Overall, the rebuttal has adequately addressed my major concerns, and I will maintain my original rating of 3/4 (Weak Accept).
Thank You to Reviewer 6QFZ
Dear Reviewer 6QFZ,
We appreciate your careful reading of our response and your thoughtful comments throughout the discussion. It is good to hear that the end-to-end results at 32K context resolved your main concern about practical speedup.
Your review helped us make the paper stronger, and we are grateful for it.
Best regards, Authors of Submission 48
Response to Reviewer 6QFZ
We sincerely appreciate the reviewer's time and effort in providing thoughtful and constructive feedback. Please find our responses to each point below.
W1. Novelty
As the reviewer noted, channel permutation itself was proposed for weight pruning, and we agree that the algorithmic novelty is limited. The contribution we intended to emphasize is the observation that the loss of 2:4 pruning on the KV cache largely comes from how channels are placed on the key side, rather than being an inherent cost of the structural constraint, which prior work (MUSTAFAR, HieraSparse) reasonably took as a given. We also observed that a single sort was sufficient, with more elaborate search providing only marginal additional gain, and that the permutation can be fixed in advance with little sensitivity to documents, tasks, or calibration data. In practice, this brings a 2:4 pattern close to the accuracy of unstructured pruning (within 0.3 points on LongBench) while being 1.21× faster than dense end-to-end at a 32K context with batch size 8 (please see our response to W2 & Q1).
W2 & Q1. End-to-end efficiency across context lengths and batch sizes
As you suggested, we measured end-to-end time across context lengths and batch sizes, including the permutation and pruning overheads (NVIDIA A100 80GB, Llama-3.1-8B-Instruct (fp16), generation length 4,096). The compared methods are dense (FlashAttention-2), unstructured (MUSTAFAR's official kernel), and CheeSe (including permutation and OA).
Prefill is nearly the same across the three methods; the gain appears in decode, which reads the entire KV cache for every token. The gain grows with the KV cache: at 32K × 8 the total time is 1.21× shorter, while at 8K × 1, where the KV cache is small, CheeSe does not provide a speedup (0.94×). We would like to add that CheeSe accelerates only the attention over the KV cache during decode, while prefill and the remaining decode operations (e.g., the MLP) are shared with dense. The 1.80× speedup of decode attention in Figure 2(b), measured at 32K × 8, therefore corresponds to 1.26× in decode and 1.21× end-to-end. The overhead added by CheeSe (permutation, OA, pruning, packing) is 1.5% of the total time and is included in the numbers below.
In terms of end-to-end throughput, CheeSe reaches 219 tok/s at 32K × 8, compared with 181 tok/s for dense.
Q2. Practical benefit of output-aware (OA) selection
As the reviewer noted, the overall gain of OA is small (+0.01 to +0.23 on the 16-task average). Most of CheeSe's recovery of the 2:4 loss comes from the channel permutation, and OA is a component added on top of it. On specific tasks, however, the gain was consistent across the evaluated models.
On the LongBench code-completion tasks (average of Lcc and RepoBench-P), the five models improve.
On the RULER word-frequency aggregation task (cwe), the three evaluated models also improve. cwe is the subtask where the largest gap to dense remains after the permutation, and OA closes more than half of that gap.
OA adds only 0.5% to the total generation time, so we suggest using it as an optional component for tasks such as code completion and aggregation, where we observed a consistent gain. We will clarify this in the revised manuscript.
Thank you again for the helpful review.