acceptodds
Under review as a conference paper at ICLR 2027

Select Twice, Freeze Once: Jacobian-Based KV Cache Selection for Multi-Turn and Long-Context LLM Serving

Abstract

The long-term deployment of multi-turn LLM systems depends on reliable and efficient handling of ever-extending context windows. As the KV cache scales linearly with sequence length, it has become a primary bottleneck for memory and throughput. This overhead is particularly detrimental to multi-turn systems, which generate a high volume of prefill and decode calls and are therefore disproportionately sensitive to the overhead of low-rank approximation techniques. Existing KV cache selection methods rely on the attention map as the primary selection signal. We find that the attention-output Jacobian motivates considering local sensitivity beyond individual attention weights when selecting entries for reuse. We introduce PrescienceKV, which combines a linear surrogate derived from a quadratic scoring proxy with windowed attention-based reranking. The surrogate constructs candidates, while reranking aggregates local value sensitivity over recent prompt queries and the first decode query to select the final top-k historical subset. Both stages run at the first decode step of each turn over the prefilled history, after which the subset is frozen and all newly generated KV entries are appended without further selection. We evaluate PrescienceKV on RULER and LongBench across five models for accuracy and efficiency. PrescienceKV reduces the selected historical KV budget by over 96% at 128K context while maintaining accuracy comparable to full attention. At capacity, PrescienceKV sustains a decode batch of 16-20 across context lengths up to 124K, where full attention's maximum batch drops to 2, yielding decode-throughput speedups up to 6.01x. In the separate per-token latency comparison at 128K, it reduces latency by 33% relative to full attention.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
KV cache, KV cache selection, LLM inference, serving, CPU offloading

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_7qvbOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes PrescienceKV, a KV-cache selection and offloading method aimed at multi-turn and long-context LLM serving. Its starting point is an attention-output Jacobian analysis: for a single query the per-key Jacobian block is Jkj=ajd (vj−y) q⊤,J_{k_j} = \frac{a_j}{\sqrt{d}}\,(v_j - y)\,q^\top , while the per-value Jacobian has operator norm ∥Jvj∥2=aj,\|J_{v_j}\|_2 = a_j ,

so attention weights exactly measure local output sensitivity to value perturbations. The authors argue this sensitivity is not captured by attention weights alone.

The method runs two stages, both only at the first decode step of each turn, after which the selected subset is frozen. Stage 1 builds candidates from a cheap linear surrogate obtained by differentiating a quadratic feature-space proxy

F2(q,k)=∑ℓ=1deqℓ(1+kℓ+12kℓ2),F_2(q,k) = \sum_{\ell=1}^{d} e^{q_\ell}\Bigl(1 + k_\ell + \tfrac{1}{2}k_\ell^{2}\Bigr),

whose derivative with respect to the keys is

∇kF2=exp⁡(q)⊙(1+k);\nabla_k F_2 = \exp(q)\odot(1+k);

two scores are then used, a signed derivative sum T and a normalized directional-agreement score S, reduced over the window by logsumexp. Stage 2 reranks the candidate union (plus recent and outlier indices) with candidate-restricted attention over a window of the first decode query and up to w_q recent prompt queries, takes the max over the window and the GQA group, and keeps the top-b historical entries per KV head. Values are offloaded to CPU except for outliers and selected entries; all newly generated KV entries are appended without further selection until the next turn's reconstruction.

Validation covers LongBench (6 tasks, up to 8,192 tokens, budget 2,048) on Llama-3.1-8B and Qwen2.5-3B; RULER (9 tasks at 65,536 and 131,072 tokens, budget 4,096) on Llama-3-8B-1M, GLM-4-9B-1M and Qwen3-8B; NIAH; multi-turn RULER-NIAH and LoCoMo; and efficiency measurements (max decode batch, per-token latency, memory) with a vLLM integration. Headline results: a 96.875% historical KV reduction at 128K with accuracy described as comparable to full attention, a decode batch of 16–20 versus 2 for full attention, decode-throughput speedups of 1.24x to 6.01x, and a 33% per-token latency reduction at 128K.

Critical strengths
  1. A real, end-to-end serving result. The method is integrated into vLLM and delivers a concrete capacity and throughput gain: at 4,096 entries per KV head it sustains a decode batch of 16–20 versus 2 for full attention, with throughput speedups of 1.24x to 6.01x (Table 1(a), Table 9). This is consequential because it shows the idea survives real serving constraints rather than only a kernel microbenchmark, and it is the strongest single reason to lean toward acceptance.

  2. The per-turn freeze-and-append schedule is a clean, transferable design point. Running both selection stages only at step 0 of a turn and appending new entries without re-selection (Section 4, Algorithm 2) removes the per-step selection cost of Quest and ShadowKV and avoids the repeated SVD cost that Figure 1 (right) identifies as the multi-turn bottleneck. Reframing when to select, rather than only what to select, is the most reusable contribution here and deserves more emphasis than the Jacobian narrative.

  3. Unusually candid scoping. The Limitations (Appendix G) and the repeated disclaimers in Sections 3.1 to 3.3 and Appendix C are honest about what the surrogate does and does not guarantee. That honesty is a genuine positive, and it is what makes the remaining disagreements tractable rather than fatal.

Critical weaknesses
  1. The "Jacobian" thesis is not carried by the deployed method. The diagnostic in Figure 1 uses the exact Jacobian Jkj=ajd (vj−y) q⊤,J_{k_j} = \frac{a_j}{\sqrt{d}}\,(v_j - y)\,q^\top , which PrescienceKV never computes, and its own caption notes that the diagnostic does not reflect the deployed surrogate's performance. Sections 3.2 and C go further, stating that the surrogate is neither a softmax approximation, nor a gradient norm, nor a bound on the key Jacobian. The problem is not the honesty but that the conceptual claim is left unsupported, and the paper's own order ablation cuts against it: in Table 5 the derivative score is no better than the undifferentiated quadratic proxy,∇KT2=0.667, 0.333versusT2=0.700, 0.367.\nabla_K T_2 = 0.667,\ 0.333 \quad\text{versus}\quad T_2 = 0.700,\ 0.367 . If differentiating the proxy does not help, the Jacobian inspiration is doing framing work rather than algorithmic work.

  2. The efficiency claims disagree across the paper's own tables. The abstract promises a 33% latency reduction relative to full attention (matching Table 1(b), 0.67x at 128K), yet Table 10 reports a single-batch decode latency of 39.66 s for PrescienceKV versus 33.68 s for full attention at 128K, that is, slower. At 64K the split is sharper still: Table 1(b) reports 0.96x (essentially a tie) while Table 10 reports 15.23 s versus 10.20 s. A reader cannot tell which configuration the headline number refers to. Separately, the 6.01x figure is a throughput ratio at each method's own maximum batch and explicitly excludes prefill (Table 9 caption); presenting it alongside a latency claim invites the misreading that it is a matched-compute speedup.

  3. The ablations credit the wrong component, and the attention-only control is not credible. In Table 7, removing the outlier heuristic costs about five points (34.59 down to 29.54), so a large share of the benefit comes from a retention heuristic rather than from the surrogate. At the same time the attention-only control collapses to 13.30, whereas published attention-based methods reach 29.61 (Quest) and 30.39 (ShadowKV) on LongBench (Table 2). A control that sits 16 points below every published attention-based method cannot be used to argue that attention-based selection is fundamentally limited. Relatedly, the multi-turn claim, which the title foregrounds, rests on the qualitative Figure 3(b): LoCoMo is never tabulated and no numerical baseline comparison is given, leaving the paper's central multi-turn promise the least evidenced part of the work.

Questions for authors
  1. Figure 1 (left) plots the exact-Jacobian diagnostic Jkj=ajd (vj−y) q⊤,J_{k_j} = \frac{a_j}{\sqrt{d}}\,(v_j - y)\,q^\top , which the method does not compute. Could you add the deployed surrogate's curve on the same axes? As it stands, the evidence that attention weights alone are insufficient is produced by a selector the paper never deploys.

  2. Why is the second-order Taylor expansion of exp(k) the appropriate proxy, and why is differentiating it preferable to scoring directly with T2T_2? Given that Table 5 shows T2T_2 is at least as good as ∇K\nabla_K T2T_2 on both models, can you exhibit any setting (budget, model, task) where the derivative form is strictly better? (This bears directly on whether the paper's core motivation is necessary.)

  3. Please reconcile Table 1(b) and Table 10. Which configuration produces the 0.67x per-token latency, and why does the single-batch decode latency in Table 10 exceed full attention at both 64K and 128K? Please also confirm the prefill latencies of 0.016 s at 8K and 0.025 s at 64K in Table 10, which look implausibly small for an 8B model.

  4. In Table 7, attention-only selection scores 13.30, while Quest and ShadowKV reach 29.61 and 30.39 with attention-based selection (Table 2). What protocol difference produces this gap? Specifically, is the attention-only control run with the identical query window, outlier policy, candidate budget, and freeze schedule as PrescienceKV?

  5. Tables 11 to 13 report auxiliary state of 730 to 972 MiB per layer, L×s MiBwithL≈32–36, s≈730–972,L \times s \ \text{MiB} \quad\text{with}\quad L \approx 32\text{--}36,\ s \approx 730\text{--}972 , comparable to the KV pool itself. Since the batch-capacity advantage is the main efficiency claim, how is this state included in the capacity comparison, and how does it scale with context length and with wqw_q?

Reviewer_h2stOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes PrescienceKV, a KV-cache selection and offloading system for long-context and multi-turn LLM inference. At the beginning of each turn, it combines the last 32 prompt queries with the first decoding query, constructs candidates using a Jacobian-inspired surrogate, and reranks them using attention. The selected historical KV subset is then frozen within the turn, while newly generated KV entries are appended. Experiments cover LongBench, RULER, and NIAH with latency and throughput.

Critical strengths

The paper implements the method in vLLM and evaluates both accuracy and serving efficiency across multiple models and context lengths. The evaluation is extensive, covering LongBench, RULER, NIAH, and LoCoMo, including question answering, synthetic retrieval, and multi-turn retrieval. The reported accuracy and throughput suggest practical potential for model serving.

Critical weaknesses
  • The central algorithm remains close to SnapKV: use a short observation window to select historical KV entries and reuse the subset during generation. The method still relies on the observed queries (the last 32 prompt tokens) being representative of subsequent information needs, which refines SnapKV's paradigm rather than fundamentally addressing the issue. The evaluation does not establish whether it remains robust when an NIAH question appears earlier in the prompt, outside the last 32 prompt tokens. Value offloading, GPU-resident outlier protection, and overlapping computation with transfers also already have precedents in ShadowKV.

  • The paper does not sufficiently establish the individual contributions of the added components. For example, the paper does not isolate the benefit of second stage attention reranking or establish the complementarity of the T/S scoring branches, while Appendix C reports nearly identical scores with and without normalization.

  • In Table 3, the average improvement over the strongest reported baseline is approximately 0.5-1.3 percentage points. These relatively small margins make a efficiency comparison important. However, every reported baseline in Table 1(b) is slower than full attention. The substantial slowdowns of sparse attention baselines relative to full attention raise concerns about baseline implementation quality and comparison fairness. The paper does not provide sufficient implementation validation or latency breakdowns to establish whether these slowdowns arise from intrinsic algorithmic costs or avoidable implementation or configuration overheads.

Questions for authors
  1. How robust is PrescienceKV when the question falls outside the last 32 prompt tokens? Please provide a controlled NIAH comparison that varies question position (far away from / near / inside the last 32 prompt tokens) while keeping the needle position and total context length fixed.

  2. Appendix C reports nearly identical scores with and without normalization. What benefit does S provide beyond T-only selection? Could you clarify what the normalization ablation changes and compare T-only against T+S under matched final KV budgets and comparable candidate-set sizes?

  3. Why are several sparse attention baselines substantially slower than full attention instead of faster? How have the authors ruled out suboptimal implementations or configuration mismatches as the cause?

Reviewer_4jJfOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

To develop a KV cache selection method without accessing attention maps, authors provided PrescienceKV, which leverages the Jacobian-inspired two-stage KV cache selection method. The authors introduce a feature product, exp(q+k), to build a proxy for the KV selection score. Then, applying second-order Taylor decomposition on the feature product produces Eq. (5), followed by its key derivatives (6), where the resulting exp(q)(1+k) is further revised to produce proxy scores Eq (7) and (8). Using these token-level scores, PrescienceKV first selects a subset of candidate tokens using Eq (8) during the prefill stage. During decoding, it combines T and S scores within a given window to identify the final candidate indices C, which are then used to construct the frozen buffer. Evaluating on two benchmark datasets, authors claim that PrescienceKV achieves comparable accuracy while reducing KV budget by over 96%.

Critical strengths
  • The proposed method has strong empirical results, where the performance is maintained with only 4% of the selected KV cache.
  • By increasing the batch size, the proposed method achieves a 6x speedup.
Critical weaknesses
  • I think the major weaknesses of this paper are the unclear connection and justification among the goal ||Y-Y_S||, the Jacobian inspiration, and Eqs (7) and (8).

  • First of all, JVP is better than the Jacobian in terms of approximating ||Y-Y_S||. For instance, suppose y=\sum_i w_i x_i. The JVP is [w_1x_1, w_2x_2, …, w_dx_d], reflecting the actual difference between Y - Y_S. The Jacobian = diag([w1, …, wd) only represents the sensitivity, which is not a signal that directly minimizes | | Y- Y_S | |. Furthermore, this JVP analysis has already been applied in the KV cache eviction domain (https://arxiv.org/abs/2601.07891).

  • Second, Section 3.2 starts by introducing exp(q+k) without justification. Then, the resulting (7) and (8) are simply the dot product and cosine similarity between exp(q) and (1+k). This paper should provide the reason for how the JVP analysis is connected to exp(q+k). If not, it is better to remove Section 3.1 and 3.2, and start with Section 3.3 and focus more on justifying the use of exp(q) and (1+k).

  • Lastly, the novelty of this work is limited, as closely related methods have already been proposed. Taking the maximum of the attention score alongside specific windows and corresponding heads in Eq. (11) is already proposed in KVzip (https://arxiv.org/abs/2505.23416). The re-ranking concept for KV cache utilization is also already proposed in FlashMemory-Deepseek-V4 (https://arxiv.org/abs/2606.09079).

  • Besides, the major gain of this paper comes from the use of outliers, but the justification is not provided. The outlier scoring means that the tokens with having low <exp(q), (1+k)> are important, where the connections between minimizing | | Y- Y_S | | or attention score itself are not provided.

Questions for authors
  • I couldn’t understand Figure 1. The left figure shows that as the budget reduces, the exact Jacobian sharply drops F1 accuracy, while the other methods maintain relatively high accuracy. What does the caption, “while Jacobian-based selection retains quality down to 0.1% of the original budget, " mean?
  • Why does HF have a better max decode batch than vLLM in Table 1? Could you explain how your method is compatible with PagedAttention?
  • Modern architectures inherently include these selection modules, including DeepSeek V3.2, GLM-5.2, MiniMax-M3, HySparse2, etc. FlashMemory (https://arxiv.org/abs/2606.09079) proposed Lookahead Sparse Attention, achieving 90% memory reduction with even better performance than the original model. What are the strengths of PrescienceKV compared to this method?
  • In Algorithm 2, C is constructed by UniqueAcrossHeads(IT,IS,LastIndices(K,wrec),Iout). After stage 2, how many indices from each index set survive in the final KV cache?
  • I think the strongest finding of this paper is the usefulness of the outliers, where the ablation is given in Table 7. Is there any explanation for why these outliers exhibit a critical role?
  • Are there any unified scores that can reflect all the scores, T, \bar{S}, and -S_h, at the same time?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.