acceptodds
Under review as a conference paper at ICLR 2027

PRISM: Request-Level Profitability Control for KV Cache Reuse

Abstract

Reusing key–value (KV) states is a central approach to reducing redundant computation in long-context language model serving. Existing serving systems cache shared prefixes to avoid repeated prefill, while modular caching and cache fusion extend reuse to structured context components. Cache storage, transfer, and sharing mechanisms further expand reuse across serving engines and compatible model variants. These mechanisms make candidate states available, but cache availability alone does not determine whether reuse is profitable for a request. Cache construction, transfer, and reuse preparation can consume the saved target computation, making compatibility, residency, and workload characteristics part of the latency decision. We introduce PRISM, which adds a request-level profitability admission layer after a candidate cache becomes available. The admission layer jointly checks cache compatibility, physical target residency, and a profiled cost margin, and routes rejected or failed requests to Native execution. Admitted requests execute a contiguous history suffix and the query against a read-only cache view, avoiding sparse cache assembly. On ARC and OpenBookQA, the resident executor reduces target-side latency by 37.6–53.9% in controlled long-history experiments, with the same observed accuracy as Native. On the paired evaluation, Tail is faster than the sparse-assembly proxy on 127/128 questions, with positive reductions persisting across fixed history bins, timing repeats, and tested tail ratios.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
KV cache reuse, LLM inference, profitability-aware admission control, cache residency, latency optimization.

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_N5b3OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes PRISM, a request-level admission layer that decides whether a request should reuse an available KV cache or fall back to Native execution. The method checks cache lineage, tensor geometry, position span, and device residency, and admits reuse when the history is long enough and the profiled reuse cost is sufficiently below Native. Admitted requests run a Tail executor that copies the resident cache prefix and recomputes the last 16% of the history together with the query, while rejected or failed requests fall back to Native. The paper evaluates the method on ARC and OpenBookQA with Qwen3-4B, reporting 37.6–53.9% lower target-side latency than Native when the cache is resident, and lower latency than a CacheBlend-style sparse-assembly proxy on 127/128 questions.

Critical strengths
  1. The paper separates the resident cost from the construction-inclusive cost of reuse. This accounting is useful, as it shows that the same reuse path can be profitable in the resident regime but slower than Native once cache construction is included.
Critical weaknesses
  1. (Quality / Significance) Under the admission contract, the reuse decision appears close to the standard prefix-caching decision, which makes the contribution of the controller difficult to distinguish from a cache hit.

    a. The compatibility check requires the exact model, tokenizer, tokenized history, geometry, and positions, so the admitted cache corresponds to KVθ(H)KV_\theta(H) and full reuse should already be lossless.

    b. In this case, recomputing the last 16% of the history may not provide additional quality benefit, and the appendix shows that the latency of Tail is within 0.23–0.41% of direct full reuse.

    c. The 37.6–53.9% reduction may therefore largely reflect the hit-versus-miss difference already provided by prefix caching in vLLM and SGLang, with identical accuracy expected by construction.

    d. The admission problem seems more challenging for approximate caches, such as non-prefix chunks or cross-variant transfer, where reuse affects quality. It would strengthen the paper to show how the gate handles such cases.

  2. (Quality) The comparisons and workloads may not fully locate the profitability boundary that matters in serving.

    a. The sparse-assembly proxy is designed for non-prefix chunks, so on an exact contiguous prefix it may perform selection and assembly that are not needed, which makes the comparison less informative about the Tail design.

    b. The natural histories are short multiple-choice prompts, repeated to reach 2.5K tokens at batch size one. In this setting, the gate rejects all natural-length requests and admits all stretched resident requests, so behavior near the boundary is not observed.

    c. Queueing, batching, memory pressure, and transfer bandwidth, which often determine reuse profitability in practice, are not modeled.

    d. The thinking-model checks use a 64-token generation cap and show near chance-level accuracy on ARC, which makes those comparisons difficult to interpret.

Questions for authors
  1. (ref Weakness 1) Given that the cache contract guarantees an exact target-native state, what does tail recomputation add over direct full reuse? In which cases would PRISM make a different decision from the hit/miss rule of a prefix cache?

  2. (ref Weakness 1) Could the admission rule be extended to approximate caches where reuse affects accuracy? How would it differ from LMCache's load-versus-prefill decision?

  3. (ref Weakness 2) Could the authors explain why the CacheBlend-style sparse assembly is an appropriate baseline for an exact contiguous prefix?

  4. (ref Weakness 2) Does Qwen3-4B-Thinking complete its reasoning within the 64-token cap? How many questions are in the 8×/16× screening set?

  5. (Presentation, minor) The abstract and conclusion mention the same accuracy as Native, while Table 7 shows 6.25% for Tail and 12.50% for Native on OpenBookQA. It would also help to report the number of pairs instead of "Wins: ALL" in Tables 1–2, and to define terms such as "sealed questions," "formal package," and "screening rows."

Reviewer_SeXzOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

PRISM admits KV-cache reuse subject to compatibility, residency, history length, and estimated latency. The paper defines a target-native cache contract, distinguishes resident execution from construction-inclusive cost, and compares a partial-recompute Tail path with Native execution and a CacheBlend-style sparse-assembly proxy. It also studies a natural-length calibration/holdout replay and longer-history operating regimes.

Critical strengths
  1. The explicit cache contract clearly describes under which circumstances reuse is valid and thus the system’s assumptions and applicability are transparent.
  2. By matching the measurements from the same experiments (Table 4) can show the benefits of reuse as well as the costs, such as a 7.19% reduction in warm-time over the sparse-assembly proxy, but a 19.14% slowdown in construction-inclusive time over Native.
Critical weaknesses
  1. The central cost gate is not established as a useful selection policy. The main experiments compare fixed execution paths, while a separate natural-length threshold search rejects reuse under the 0.90 cost criterion. The 700-token threshold falls within an untested gap in the evaluated history lengths. All reported routing is consistent with a simple residency-plus-length rule, without demonstrating additional benefit from the cost test.

  2. The executor comparison does not establish why suffix recomputation is needed for an exact compatible history cache. A separate runtime measurement shows Tail performing nearly identically to query-only full reuse, but under different requests and conditions. Appendix A describes this as an already-materialized-cache comparison, whereas Fig. 5b labels it cold-path. A matched comparison is needed to establish Tail's additional value.

Questions for authors
  1. [Suggestion] Does the frozen controller reduce total latency, including decision overhead, relative to Native and a residency-plus-length rule on a held-out mixed workload?
  2. [Clarification] What motivates suffix recomputation, and what timing boundary and runtime support the comparison with query-only full reuse?
Reviewer_X147OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

Cache construction, transfer, and reuse preparation can consume the saved target computation, making compatibility, residency, and workload characteristics part of the latency decision. This paper introduces PRISM, which adds a request-level profitability admission layer after a candidate cache becomes available. Experiments show the performance.

Critical strengths
  1. KV-cache reuse in LLM serving is an important topic. The request-level profitability control is important.

  2. The warm latency and cold total latency are evaluated based on different situation.

  3. Experiments are provided.

Critical weaknesses
  1. The mechanism of the proposed tail method is unclear. How does the method compare to full-reuse the kv-cache?

  2. The performance of the request controller is unclear. The comparison with existing KV caching systems is not provided.

  3. Experiments are only on qwen-3-4B with single turn QA task.

Questions for authors

The overall writing needs huge improvement. I would encourage authors read through the paper to better understand the content and improve the quality.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.