acceptodds
Under review as a conference paper at ICLR 2027

PARSEE: Paging-Aware KV Cache Compression with Shared Semantic Support

Abstract

Long-context language-model serving requires large key-value (KV) caches, consuming GPU memory and limiting request concurrency. Although token eviction reduces logical cache size, heterogeneous per-head demands and page granularity can prevent these reductions from translating into reusable physical capacity. To address this challenge, we propose PARSEE, a paging-aware KV cache compression framework with shared semantic support. PARSEE is based on the observation that unequal retained counts among members sharing a page group inflate storage costs, and that shared semantic scores can coordinate these counts while preserving independent token identities. It selects shared semantic and head-specific question evidence through disjoint quotas, then allocates indivisible pages by marginal utility under a physical budget. Furthermore, a paged execution backend packs selected entries, transfers page ownership, and reclaims dense blocks, enabling batched attention over request–head groups with different retained lengths. Extensive experiments on five representative models from the Qwen and Llama families, covering dense, grouped-query, hybrid, and mixture-of-experts architectures, show that PARSEE retains near-lossless SCBench performance on Qwen3.8-27B while compressing 90% of prefill-context KV. Its Qwen3-8B backend achieves up to the throughput of uncompressed vLLM inference.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, KV Cache Compression

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_uW4uOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PARSEE to translate logical KV cache compression into reclaimable physical memory. It combines shared semantic scores with head-specific attention to coordinate retention demands within page groups, then allocates pages under a physical memory budget. A paged backend packs selected KV entries, updates page tables, and reclaims the original blocks.

Critical strengths
  1. The paper addresses a meaningful systems problem. Equal numbers of retained KV entries can incur different physical storage costs because of unequal member lengths and page rounding. Jointly considering token selection and page allocation makes compression more relevant to practical memory capacity and serving concurrency.

  2. Accuracy remains strong under aggressive compression. With 90% prefill-context KV compression, PARSEE preserves good SCBench performance across several models.

Critical weaknesses
  1. Compatibility with existing serving optimizations remains uncertain. Query-conditioned KV selection, page rearrangement, and ownership transitions may impose additional constraints on prefix caching and CUDA Graph execution. Although the authors discuss a possible cache-reuse design, the experiments disable prefix caching and use eager execution. Compatibility and performance when combining these techniques therefore remain unverified.

  2. The complex pipeline may increase per-request latency and TTFT. The method involves feature extraction, semantic scoring, dual-quota selection, page allocation, and KV packing. Since selection depends on the current question, these operations may need to be repeated for new requests, introducing additional overhead.

Questions for authors

N/A

3 comments

Thank you for recognizing the importance of physical storage costs and PARSEE's strong SCBench accuracy under aggressive compression.

1. Is PARSEE compatible with prefix caching and CUDA Graph execution? (W1)

PARSEE can be integrated with prefix caching and CUDA Graph execution, provided that shared-cache ownership and graph-replay constraints are respected. Prefix caching can retain a persistent query-agnostic source cache while PARSEE constructs separate query-specific compressed working sets (Appendix H.3). CUDA Graph execution requires stable captured buffers and shapes, with dynamic allocation and ownership changes handled outside replay; Appendix F.7.3 (Table 25) provides kernel-level replay evidence.

To examine their combined performance, we reuse the online serving workload above with a shared 32K prefix, different questions, 512 output tokens, and concurrency four, enabling prefix caching and CUDA Graphs separately and jointly. Prefix caching uses a warm cache; timing includes query-specific scoring, and memory includes both persistent shared storage and per-query destinations. Each cell reports Full KV / PARSEE.

Prefix cachingCUDA GraphsTTFT P95 (s)Output tokens/sPeak used GPU memory (GB)
OffOff28.0 / 29.228.6 / 38.937.5 / 40.2
OnOff8.0 / 11.036.3 / 40.240.9 / 42.0
OffOn27.8 / 29.029.1 / 38.538.0 / 40.7
OnOn7.8 / 10.836.6 / 40.540.5 / 42.5

With both optimizations enabled, PARSEE has higher throughput than Full KV, while TTFT remains higher.

2. How much latency does the pipeline add, especially for a new question and the first output token? (W2)

We already report these costs in Appendix F.7.2 (Tables 20–23), and include online serving throughput and TTFT in Appendix F.6 (Table 18). For 32K contexts, PARSEE spends 1.7 s on scoring and 1.3 s on selection and packing, with a total batch latency of 56.9 s, compared with 59.0 s for TANGRAM+SnapKV. The benefit depends on the workload: TANGRAM+SnapKV is faster on shorter contexts, and PARSEE has higher TTFT than Full KV at concurrency one. At concurrency eight, however, PARSEE reaches 273.4 tokens/s, compared with 80.2 tokens/s for Full KV. These online measurements include query-conditioned selection and compilation for each request, so the reported gains already account for this extra work.

Thank you again for your thoughtful review! We would appreciate any feedback on whether our rebuttal and additional experiments address your concerns. If any questions remain, we would be happy to clarify or discuss them further!

With less than 24 hours remaining in the discussion period, we would appreciate any feedback on whether our rebuttal and additional experiments address your concerns. We remain available to clarify any outstanding questions. Thank you again for your time and review!

Reviewer_yWJzOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposed PARSEE, a framework that addresses the gap between logical token reduction and actual GPU memory savings in long-context LLM serving with KV cache compression. The authors identify physical memory inefficiencies caused by imbalanced retention lengths across layer-head groups and page allocation granularity. The proposed PARSEE combines a shared semantic reference with head-specific attention evidence to coordinate token selection, then uses a budget-aware physical compiler and cache management backend to allocate and reclaim memory efficiently. Experiments across 5 models demonstrate promising performance preservation under aggressive compression, with improvements in memory utilization and decoding throughput.

Critical strengths
  1. The paper identifies an important systems-level limitation of existing KV-cache compression methods and proposed PARSEE to explicitly modeling page allocation and retention imbalance to address the limitation.

  2. The combination of shared semantic information and head-specific attention evidence is well motivated. And the paper shows some analysis and controlled experiments to provide useful evidence for the effectiveness of coordinated selection.

  3. The evaluation covers various model architectures, including dense, hybrid, and MoE configurations, and demonstrates promising quality-capacity trade-offs. The paper also includes targeted ablations and per-task results that help distinguish the contributions of different selection components.

Critical weaknesses
  1. The scorer-inclusive comparisons do not demonstrate a consistent latency advantage over TANGRAM. The paper should distinguish backend acceleration from end-to-end performance more clearly and provide a consistent online comparison that includes all compression and cache-management overheads.

  2. The independent contribution of the physical compiler is not sufficiently isolated from the token-selection strategy. While coordinated semantic selection reduces retention imbalance, the final allocation still incurs substantial overhead from head-specific attention evidence. The reported ablation without physical projection shows only a small quality difference, without sufficient statistical evidence to establish the compiler's benefit. A comparison using identical selected candidates and physical memory budgets, but different allocation strategies such as uniform truncation or equal page allocation, would better demonstrate the value of budget-aware compilation.

Questions for authors
  1. How much of the retrieval performance depends on explicit identifier protection rather than semantic coordination? Can the authors provide more detailed task-level results or ablations that separate these effects?

2.What are the peak memory footprint during compression, and how do they affect maximum supported concurrency and request admission under a fixed GPU memory budget?

  1. How sensitive are the efficiency gains to heterogeneous request lengths, different grouping configurations, and cache reuse across queries? Does the advantage persist when physical memory usage and output quality are matched across methods?
6 comments

Thank you for recognizing the physical-memory problem, the motivation for coordinated selection, and the breadth of our evaluation and ablations.

1. Is there a consistent online advantage over TANGRAM after including all overheads? (W1)

To provide a fair comparison of backend execution, our main throughput experiments use precomputed PARSEE selections and exclude their online scoring and selection time (Appendix C.6). This setting was motivated by TANGRAM's use of offline-calibrated head budgets.

Following your suggestion, we extend the backend setup in Appendix C.6 (Table 10) to E2E evaluation, using 32K raw contexts, 512 output tokens, and four closed-loop requests on one A100-80GB. All online scoring, selection, compilation, packing, and reclamation costs are included, with compression preceding first-token emission and contributing to TTFT and request latency.

MethodCompression overhead, mean (s)TTFT P95 (s)Request latency P95 (s)Output tokens/s
TANGRAM+KeyDiff2.030.563.235.9
TANGRAM+SnapKV1.730.062.036.7
TriAttention1.129.964.535.2
PARSEE2.929.260.038.0

The table above reports E2E serving performance, including online scoring, compression, and cache reclamation. PARSEE has the highest throughput and lowest P95 request latency among the compared methods in this setting. Also notably, PARSEE achieves substantially better task performance than these methods while maintaining comparable efficiency.

2. What benefit does the physical compiler provide when selection and the physical budget are fixed? (W2)

We agree that a comparison with identical pre-compilation candidates and physical budgets is necessary to isolate the compiler's contribution. The comparison below isolates this contribution.

On a separate allocation cohort, we fix candidates, scores, protection, member ordering, and layout, and vary only the allocation strategy under a 460.8-MiB context-KV cap per request.

Allocation strategyCharged context KV (MiB)Candidate recall (%)Free-candidate recall (%)SCBench macroΔ vs. equal pages
Uniform truncation460.887.1085.6343.60−0.20
Equal-page allocation460.887.6586.2543.80Reference
PARSEE compiler460.889.5488.3544.10+0.30

At the same budget, PARSEE increases candidate recall by 1.89 percentage points over equal-page allocation, with a small SCBench gain of 0.30 points.

Appendix A.2 establishes utility optimality under its stated assumptions, while Appendix F.7.3 (Table 24) shows that grouped pages reduce storage costs for PARSEE's fixed selections compared with uniform-capacity pages.

3. How much of retrieval quality comes from identifier protection rather than semantic coordination? (Q1)

We have already included anchor-removal and protection-mechanism ablations in Appendix F.1. Table 12 shows that removing identifier anchors mainly harms KV retrieval, while Appendix F.2 (Table 13) shows an additional semantic-reference benefit with protection fixed. To further verify whether the semantic benefit persists without identifier protection, we conduct the following crossed ablation.

We follow the Qwen3-8B SCBench setup in Appendix F.2 (Table 13), crossing reference choice with anchors on/off at the same physical budget. Sinks and quotas remain fixed, and freed anchor slots return to the free-entry budget. Each cell reports anchor-on / anchor-off scores.

SCBench taskMatched shared attention, on / offMatched shared semantics, on / offNative PARSEE, on / off
KV retrieval88.00 / 27.0091.50 / 31.0077.60 / 32.00
Prefix retrieval80.00 / 75.0084.00 / 79.0046.60 / 80.00
Variable tracking28.00 / 28.0029.80 / 29.8032.89 / 30.00
Mixed retrieval47.00 / 44.0049.50 / 46.5027.17 / 47.00
QA33.50 / 33.0034.80 / 34.3039.41 / 34.50
Choice44.00 / 43.5045.00 / 44.5073.61 / 44.50
Many-shot24.00 / 23.8025.20 / 25.0042.59 / 24.80
Summarization19.50 / 19.3820.10 / 19.9835.55 / 19.88
RepoQA24.08 / 21.5824.02 / 21.5230.00 / 20.95
Task macro43.12 / 35.0344.88 / 36.8445.05 / 37.07

For native PARSEE, anchors raise KV retrieval from 32.00 to 77.60 and the task macro from 37.07 to 45.05, although their effects vary across tasks, with lower prefix- and mixed-retrieval scores. Matched semantics improves over shared attention by about 1.8 macro points with anchors both on and off, indicating a semantic benefit beyond identifier protection.

4. What is the compression-time peak memory, and how does it constrain admission and concurrency? (Q2)

Appendix C.5 and H.4 explain why compression can temporarily increase memory usage, while Appendix F.6 (Table 18) reports generation-phase KV capacity. The comparison below considers normal admission scheduling with a 67 GB per-device memory limit, the same 32K-context and 512-output-token workload, and common scheduling rules and latency constraints.

MethodPeak used GPU memory at maximum concurrency (GB)Generation-phase KV capacity at concurrency four (GB)Maximum sustainable concurrency
Full KV64.918.2810
TANGRAM+KeyDiff65.82.1624
TANGRAM+SnapKV65.52.1624
TriAttention65.22.1624
PARSEE65.22.0824

Under the same memory budget and latency constraints, PARSEE supports 2.4× the maximum sustainable concurrency of Full KV, matching the other compression methods while using slightly less generation-phase KV memory at concurrency four.

5. Do the benefits persist with heterogeneous lengths, different groupings, cache reuse, and matched memory and quality? (Q3)

We examine grouping-dependent storage costs in Appendix D.1, input-length sensitivity in Appendix F.7.1 (Table 19), and quality–capacity–throughput trade-offs in Appendix F.6 (Table 18). These results show that grouping affects physical storage overhead, backend throughput gains persist across the additional homogeneous workloads, and online serving benefits depend on compression ratio and concurrency. Appendix H.3 discusses a two-level cache-reuse design that separates persistent shared storage from query-specific working sets.

To further examine serving sensitivity and efficiency under common memory and quality requirements, we add the comparisons below, covering mixed request lengths, grouping choices, and shared-prefix reuse.

For request-length sensitivity, we reuse the model and backend settings of Table 10 and the scorer-inclusive online protocol above, changing the input/output lengths to the listed mixtures. Input and output mixtures are independently 50:50, with the same arrival trace and memory limit across methods.

Request workloadMethodTTFT P95 (s)Request latency P95 (s)Output tokens/sPeak used GPU memory (GB)
Homogeneous 18K / 512 outputTANGRAM+SnapKV13.045.049.030.2
Homogeneous 18K / 512 outputPARSEE13.545.548.530.8
50:50 4K/32K / 512 outputTANGRAM+SnapKV30.262.643.039.5
50:50 4K/32K / 512 outputPARSEE29.460.644.240.2
50:50 4K/32K / 128 or 2048 outputTANGRAM+SnapKV32.0141.044.040.4
50:50 4K/32K / 128 or 2048 outputPARSEE31.6137.045.041.1

PARSEE improves throughput over TANGRAM+SnapKV by 2.3–2.8% on the mixed workloads, with lower P95 request latency, but is slightly slower on the homogeneous workload.

For grouping sensitivity, we follow the fixed-support comparison in Appendix D.1, replacing the grouping alternatives with the four layouts below under a common 460.8-MiB context cap. Quality follows the Qwen3-8B protocol in Appendix F.2; throughput uses the online workload above. Metadata is accounted for separately.

LayoutCharged context KV (MiB)Candidate recall (%)SCBench macroMetadata (MiB)Output tokens/s
L1H1B16460.893.1045.158.036.4
L1H2B16460.891.2045.094.037.5
L2H2B16460.889.5445.052.038.0
L2H4B16460.886.7044.721.037.4

The default L2H2B16 layout has the highest displayed throughput, with a small quality reduction relative to finer grouping. The layout comparison illustrates the trade-off between grouping granularity, quality, and execution cost.

For the memory/quality comparison, we follow Appendix F.6's separate quality and serving evaluations, using the online workload above and adding the compression baselines. We impose a common 460.8-MiB context cap, the same total-memory limit, and a quality threshold of 44.00; this tests a common quality requirement rather than exact quality matching.

MethodCharged context KV (MiB)Peak used GPU memory (GB)SCBench macroMemory/quality criteria met?Output tokens/s
TANGRAM+KeyDiff460.839.827.76Memory yes / quality no35.9
TANGRAM+SnapKV460.839.526.35Memory yes / quality no36.7
TriAttention460.839.227.13Memory yes / quality no35.2
PARSEE460.840.245.05Memory yes / quality yes38.0

All four methods satisfy the serving memory limit, but only PARSEE exceeds the benchmark quality threshold of 44.00. The table therefore shows higher benchmark quality alongside competitive online efficiency, rather than a throughput comparison between quality-matched operating points.

For cache reuse, we reuse the online workload above, replacing independent contexts with a shared 32K prefix and different questions, and toggle prefix caching and CUDA Graphs. Prefix-caching-on settings use a warm cache. Scoring remains query-specific, and memory includes the persistent source cache and per-query destinations. Each cell reports Full KV / PARSEE.

Prefix cachingCUDA GraphsTTFT P95 (s)Output tokens/sPeak used GPU memory (GB)
OffOff28.0 / 29.228.6 / 38.937.5 / 40.2
OnOff8.0 / 11.036.3 / 40.240.9 / 42.0
OffOn27.8 / 29.029.1 / 38.538.0 / 40.7
OnOn7.8 / 10.836.6 / 40.540.5 / 42.5

The throughput advantage narrows with prefix caching. These comparisons characterize performance under the listed configurations; they do not establish full serving compatibility.

Reviewer_ah6nOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

PARSEE selects KV entries using attention scores and semantic similarity while accounting for the pages needed to store them. It then packs the retained entries and releases the original cache blocks so the freed memory can be reused.

Critical strengths

S1. Practical relevance: While prior work has mainly focused on which KV entries to retain, this paper also considers their physical page costs. The implementation includes compaction and memory reclamation, helping translate KV selection into usable GPU memory savings.

S2. Analysis and experiments: The paper analyzes storage waste from uneven per-head KV counts and evaluates five models, with ablations and controls to examine the proposed designs.

Critical weaknesses

W1. Limited methodological novelty: As I understand it, PARSEE combines attention- and semantic-similarity-based KV selection with page-level memory allocation. While accounting for physical page costs during selection distinguishes it from prior work, the overall contribution appears to lie more in combining existing techniques and implementing the system than in introducing a new selection principle.

W2. Runtime comparison: The claimed maximum throughput gain of 5.28× is relative to uncompressed vLLM. However, as I understand it, this experiment excludes PARSEE’s scoring and selection time while including these costs for TANGRAM and TriAttention. Please clarify whether the speed advantage holds when these costs are included for all methods.

Questions for authors

See weakness

3 comments

Thank you for recognizing the practical relevance of physical memory reclamation and our analysis and five-model evaluation.

1. What is the methodological contribution beyond combining existing selection and allocation techniques? (W1)

In Appendix A.3–A.5, we establish our theoretical contributions and their implications for page-aware selection. Our contribution mainly concerns the coupling of selection and physical representation. We bound semantic count imbalance under shared scores and distinguish the capacity needed to represent a support from the budget needed for the allocator to preserve it, accounting for synchronization, rounding, and allocation competition. This yields our selection principle: coordinate retained counts while preserving independent token identities, then allocate pages under a physical budget.

We further show that, with protection, quotas, layout, and compiler fixed, matched semantic references improve quality over shared attention, while shuffling semantic positions degrades quality despite similar coordination in Appendix F.2. These controls support the effectiveness of our semantic reference, complementing the structural guarantees. The contribution lies more in coupling selection with physical representation.

2. Does the throughput advantage remain when scoring and selection are included for every method? (W2 / Questions)

The main throughput experiments use precomputed selections to compare backend execution under fixed selections; TANGRAM uses offline-calibrated head budgets. Offline budget calibration and per-request scoring are distinct costs: TANGRAM and TriAttention retain native inline work in this comparison, as discussed in Appendix C.6.

Although Appendix F.7.2 includes scorer-inclusive common-runtime comparisons, these measurements retain dense source KV.

Following your suggestion, we extend the backend setup in Appendix C.6 (Table 10) to E2E evaluation, using 32K raw contexts, 512 output tokens, and four closed-loop requests on one A100-80GB. All online scoring, selection, compilation, packing, and reclamation costs are included, with compression preceding first-token emission and contributing to TTFT and request latency.

MethodCompression overhead, mean (s)TTFT P95 (s)Request latency P95 (s)Output tokens/s
TANGRAM+KeyDiff2.030.563.235.9
TANGRAM+SnapKV1.730.062.036.7
TriAttention1.129.964.535.2
PARSEE2.929.260.038.0

With all online costs included, the table above shows that PARSEE has the highest throughput and lowest P95 request latency among the compared methods in this setting. Notably, PARSEE achieves substantially better task performance than these methods while maintaining comparable efficiency.

Thanks for the end-to-end results, which address my concern. The speed gain is small (about 3.5%), and the 5.28× figure seems to rely on a slow Full KV baseline (Table 10). Still, the quality results are reasonably good, so I’ll raise my score to 3. In my view, the paper is still borderline.

Thank you for raising your score and acknowledge our contributions! Compared with the vanilla vLLM, PARSEE achieves 3.41× online throughput at concurrency eight , which is 273.4 vs. 80.2 tokens/s (You could find this in Appendix F.6, Table 18). As our primary focus is KV-cache compression, the main benefit is substantially reduced KV memory: the generation-phase KV pool decreases from 9.20 to 1.68 GB, an 81.7% reduction, while preserving strong task quality. We will emphasize these benefits more clearly in the revised manuscript. We also welcome any further questions or discussion!

Comments on the paper

2 comments

Dear Reviewers,

The authors have provided their rebuttal, but no follow-up comments have been posted yet. As the author-reviewer discussion period is ending soon, please review and acknowledge the rebuttal, and engage in further discussion if any concerns remain unresolved.

AC

AuthorsOpenReview ↗

Summary of the Author–Reviewer Discussion Phase

Dear Area Chairs, Senior Area Chairs, Program Chairs, and Reviewers,

Thank you all for the discussion. During the discussion period, all the three reviewers recognized the practical value of accounting for physical page costs in KV selection. ah6n highlighted our memory reclamation experiments, yWJz found coordinated selection well motivated, and both noted the breadth of the evaluation and ablations. uW4u emphasized that our method has strong performance, even under aggressive compression.

We summarize the main questions and how they shaped the revision below. We mark all changes in the revised manuscript in purple.

  • What is new about page-aware selection?

    Reviewer ah6n asked whether PARSEE introduces a selection principle beyond combining existing scoring and allocation techniques. We clarified in Section 3 why heads sharing a physical page benefit from coordinating how many entries they retain, even when each head needs different tokens. Appendix A.3–A.5 derives how count imbalance, page rounding, and competition for pages affect which entries survive under a fixed budget. We also pointed to an existing experiment in Appendix F.2, where shuffling the semantic scores across positions preserves similar count coordination but lowers quality, showing why the token identities still matter.

  • How much speedup remains when all online costs are counted?

    Reviewers ah6n and yWJz questioned whether the reported 5.28× maximum carried over from the fixed-selection backend benchmark to end-to-end serving. We added a 32K comparison that charges each method for its online scoring, selection, packing, and reclamation. PARSEE delivers 38.0 output tokens/s versus 36.7 for TANGRAM+SnapKV, a 3.5% gain (Section 5.4, Table 7). In response to ah6n’s concern about the Full KV baseline, we highlighted a separate 4K comparison. At concurrency eight, PARSEE delivers 3.41× the throughput of uncompressed vLLM while reducing the generation-phase KV pool from 9.20 to 1.68 GiB (Table 21).

  • What do the compiler and identifier anchors each contribute?

    At reviewer yWJz’s suggestion, we held candidates and memory fixed and varied only the allocator. Our compiler improved candidate recall by 1.89 percentage points and SCBench by 0.30 points over equal page allocation (Table 6). A second experiment found that the semantic reference adds about 1.8 SCBench points over shared attention with anchors either on or off (Table 32).

  • How sensitive are the gains to memory limits and workload?

    Reviewer yWJz asked about peak memory, admission, grouping, and mixed request lengths. Under a 67 GB device limit, PARSEE peaked at 65.2 GB and sustained 24 concurrent requests, matching the other compression methods versus 10 for Full KV (Table 29). Finer grouping slightly improved quality at a throughput cost (Table 31). PARSEE was 2.3–2.8% faster than TANGRAM+SnapKV on the mixed workloads and slightly slower on the homogeneous 18K workload (Table 33). Only PARSEE met the common SCBench threshold under the same KV cap, so throughput at exactly matched quality remains open (Table 30).

  • What does a new question cost when the prefix is reused?

    Reviewers uW4u and yWJz raised cache reuse, and uW4u specifically asked about CUDA Graphs and first-token latency. We tested prefix caching and CUDA Graphs separately and together. Appendix F.7.2 (Tables 23–26) includes per-request scoring and compression in the timing. At 32K, PARSEE spends 1.7 s on scoring and 1.3 s on selection and packing, bringing total batch latency to 56.9 s versus 59.0 s for TANGRAM+SnapKV. TANGRAM+SnapKV is faster on shorter contexts. In the separate 4K serving test, PARSEE has lower TTFT than Full KV at concurrency eight. It delivers 273.4 versus 80.2 output tokens/s with these costs included (Appendix F.6, Table 21).

Reviewer feedback during the discussion

During the discussion period, reviewer ah6n said our response addressed their concern and that they would raise their score to 3. Reviewer yWJz said our response addressed their concerns and resolve the new question, so their rating remained positive.

Reviewer uW4u has yet to reply, but we believe our response addresses both of Reviewer uW4u's concerns. First, they asked whether PARSEE is compatible with prefix caching and CUDA Graph execution. We adapted PARSEE to support both and tested them together. Second, they asked whether PARSEE adds too much per-request latency and time to first token. Our experiments in the main text and appendix already show that the throughput gains can outweigh this overhead. At 4K concurrency eight, PARSEE reaches 273.4 tokens/s, compared with 80.2 tokens/s for Full KV.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.