- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes PrescienceKV, a KV-cache selection and offloading method aimed at multi-turn and long-context LLM serving. Its starting point is an attention-output Jacobian analysis: for a single query the per-key Jacobian block is while the per-value Jacobian has operator norm
so attention weights exactly measure local output sensitivity to value perturbations. The authors argue this sensitivity is not captured by attention weights alone.
The method runs two stages, both only at the first decode step of each turn, after which the selected subset is frozen. Stage 1 builds candidates from a cheap linear surrogate obtained by differentiating a quadratic feature-space proxy
whose derivative with respect to the keys is
two scores are then used, a signed derivative sum T and a normalized directional-agreement score S, reduced over the window by logsumexp. Stage 2 reranks the candidate union (plus recent and outlier indices) with candidate-restricted attention over a window of the first decode query and up to w_q recent prompt queries, takes the max over the window and the GQA group, and keeps the top-b historical entries per KV head. Values are offloaded to CPU except for outliers and selected entries; all newly generated KV entries are appended without further selection until the next turn's reconstruction.
Validation covers LongBench (6 tasks, up to 8,192 tokens, budget 2,048) on Llama-3.1-8B and Qwen2.5-3B; RULER (9 tasks at 65,536 and 131,072 tokens, budget 4,096) on Llama-3-8B-1M, GLM-4-9B-1M and Qwen3-8B; NIAH; multi-turn RULER-NIAH and LoCoMo; and efficiency measurements (max decode batch, per-token latency, memory) with a vLLM integration. Headline results: a 96.875% historical KV reduction at 128K with accuracy described as comparable to full attention, a decode batch of 16–20 versus 2 for full attention, decode-throughput speedups of 1.24x to 6.01x, and a 33% per-token latency reduction at 128K.
Critical strengths
-
A real, end-to-end serving result. The method is integrated into vLLM and delivers a concrete capacity and throughput gain: at 4,096 entries per KV head it sustains a decode batch of 16–20 versus 2 for full attention, with throughput speedups of 1.24x to 6.01x (Table 1(a), Table 9). This is consequential because it shows the idea survives real serving constraints rather than only a kernel microbenchmark, and it is the strongest single reason to lean toward acceptance.
-
The per-turn freeze-and-append schedule is a clean, transferable design point. Running both selection stages only at step 0 of a turn and appending new entries without re-selection (Section 4, Algorithm 2) removes the per-step selection cost of Quest and ShadowKV and avoids the repeated SVD cost that Figure 1 (right) identifies as the multi-turn bottleneck. Reframing when to select, rather than only what to select, is the most reusable contribution here and deserves more emphasis than the Jacobian narrative.
-
Unusually candid scoping. The Limitations (Appendix G) and the repeated disclaimers in Sections 3.1 to 3.3 and Appendix C are honest about what the surrogate does and does not guarantee. That honesty is a genuine positive, and it is what makes the remaining disagreements tractable rather than fatal.
Critical weaknesses
-
The "Jacobian" thesis is not carried by the deployed method. The diagnostic in Figure 1 uses the exact Jacobian which PrescienceKV never computes, and its own caption notes that the diagnostic does not reflect the deployed surrogate's performance. Sections 3.2 and C go further, stating that the surrogate is neither a softmax approximation, nor a gradient norm, nor a bound on the key Jacobian. The problem is not the honesty but that the conceptual claim is left unsupported, and the paper's own order ablation cuts against it: in Table 5 the derivative score is no better than the undifferentiated quadratic proxy, If differentiating the proxy does not help, the Jacobian inspiration is doing framing work rather than algorithmic work.
-
The efficiency claims disagree across the paper's own tables. The abstract promises a 33% latency reduction relative to full attention (matching Table 1(b), 0.67x at 128K), yet Table 10 reports a single-batch decode latency of 39.66 s for PrescienceKV versus 33.68 s for full attention at 128K, that is, slower. At 64K the split is sharper still: Table 1(b) reports 0.96x (essentially a tie) while Table 10 reports 15.23 s versus 10.20 s. A reader cannot tell which configuration the headline number refers to. Separately, the 6.01x figure is a throughput ratio at each method's own maximum batch and explicitly excludes prefill (Table 9 caption); presenting it alongside a latency claim invites the misreading that it is a matched-compute speedup.
-
The ablations credit the wrong component, and the attention-only control is not credible. In Table 7, removing the outlier heuristic costs about five points (34.59 down to 29.54), so a large share of the benefit comes from a retention heuristic rather than from the surrogate. At the same time the attention-only control collapses to 13.30, whereas published attention-based methods reach 29.61 (Quest) and 30.39 (ShadowKV) on LongBench (Table 2). A control that sits 16 points below every published attention-based method cannot be used to argue that attention-based selection is fundamentally limited. Relatedly, the multi-turn claim, which the title foregrounds, rests on the qualitative Figure 3(b): LoCoMo is never tabulated and no numerical baseline comparison is given, leaving the paper's central multi-turn promise the least evidenced part of the work.
Questions for authors
-
Figure 1 (left) plots the exact-Jacobian diagnostic which the method does not compute. Could you add the deployed surrogate's curve on the same axes? As it stands, the evidence that attention weights alone are insufficient is produced by a selector the paper never deploys.
-
Why is the second-order Taylor expansion of exp(k) the appropriate proxy, and why is differentiating it preferable to scoring directly with ? Given that Table 5 shows is at least as good as on both models, can you exhibit any setting (budget, model, task) where the derivative form is strictly better? (This bears directly on whether the paper's core motivation is necessary.)
-
Please reconcile Table 1(b) and Table 10. Which configuration produces the 0.67x per-token latency, and why does the single-batch decode latency in Table 10 exceed full attention at both 64K and 128K? Please also confirm the prefill latencies of 0.016 s at 8K and 0.025 s at 64K in Table 10, which look implausibly small for an 8B model.
-
In Table 7, attention-only selection scores 13.30, while Quest and ShadowKV reach 29.61 and 30.39 with attention-based selection (Table 2). What protocol difference produces this gap? Specifically, is the attention-only control run with the identical query window, outlier policy, candidate budget, and freeze schedule as PrescienceKV?
-
Tables 11 to 13 report auxiliary state of 730 to 972 MiB per layer, comparable to the KV pool itself. Since the batch-capacity advantage is the main efficiency claim, how is this state included in the capacity comparison, and how does it scale with context length and with ?