- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes PARSEE to translate logical KV cache compression into reclaimable physical memory. It combines shared semantic scores with head-specific attention to coordinate retention demands within page groups, then allocates pages under a physical memory budget. A paged backend packs selected KV entries, updates page tables, and reclaims the original blocks.
Critical strengths
-
The paper addresses a meaningful systems problem. Equal numbers of retained KV entries can incur different physical storage costs because of unequal member lengths and page rounding. Jointly considering token selection and page allocation makes compression more relevant to practical memory capacity and serving concurrency.
-
Accuracy remains strong under aggressive compression. With 90% prefill-context KV compression, PARSEE preserves good SCBench performance across several models.
Critical weaknesses
-
Compatibility with existing serving optimizations remains uncertain. Query-conditioned KV selection, page rearrangement, and ownership transitions may impose additional constraints on prefix caching and CUDA Graph execution. Although the authors discuss a possible cache-reuse design, the experiments disable prefix caching and use eager execution. Compatibility and performance when combining these techniques therefore remain unverified.
-
The complex pipeline may increase per-request latency and TTFT. The method involves feature extraction, semantic scoring, dual-quota selection, page allocation, and KV packing. Since selection depends on the current question, these operations may need to be repeated for new requests, introducing additional overhead.
Questions for authors
N/A
3 comments
Thank you again for your thoughtful review! We would appreciate any feedback on whether our rebuttal and additional experiments address your concerns. If any questions remain, we would be happy to clarify or discuss them further!
With less than 24 hours remaining in the discussion period, we would appreciate any feedback on whether our rebuttal and additional experiments address your concerns. We remain available to clarify any outstanding questions. Thank you again for your time and review!
Thank you for recognizing the importance of physical storage costs and PARSEE's strong SCBench accuracy under aggressive compression.
1. Is PARSEE compatible with prefix caching and CUDA Graph execution? (W1)
PARSEE can be integrated with prefix caching and CUDA Graph execution, provided that shared-cache ownership and graph-replay constraints are respected. Prefix caching can retain a persistent query-agnostic source cache while PARSEE constructs separate query-specific compressed working sets (Appendix H.3). CUDA Graph execution requires stable captured buffers and shapes, with dynamic allocation and ownership changes handled outside replay; Appendix F.7.3 (Table 25) provides kernel-level replay evidence.
To examine their combined performance, we reuse the online serving workload above with a shared 32K prefix, different questions, 512 output tokens, and concurrency four, enabling prefix caching and CUDA Graphs separately and jointly. Prefix caching uses a warm cache; timing includes query-specific scoring, and memory includes both persistent shared storage and per-query destinations. Each cell reports Full KV / PARSEE.
With both optimizations enabled, PARSEE has higher throughput than Full KV, while TTFT remains higher.
2. How much latency does the pipeline add, especially for a new question and the first output token? (W2)
We already report these costs in Appendix F.7.2 (Tables 20–23), and include online serving throughput and TTFT in Appendix F.6 (Table 18). For 32K contexts, PARSEE spends 1.7 s on scoring and 1.3 s on selection and packing, with a total batch latency of 56.9 s, compared with 59.0 s for TANGRAM+SnapKV. The benefit depends on the workload: TANGRAM+SnapKV is faster on shorter contexts, and PARSEE has higher TTFT than Full KV at concurrency one. At concurrency eight, however, PARSEE reaches 273.4 tokens/s, compared with 80.2 tokens/s for Full KV. These online measurements include query-conditioned selection and compilation for each request, so the reported gains already account for this extra work.