- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes RECAP-KV, a query-agnostic KV cache eviction method. It approximates KVzip’s reconstruction-based importance scores without an additional Transformer forward pass or a trained predictor. RECAP-KV constructs virtual queries from cached hidden states. It uses reference curves to allocate the retention budget, then applies Lookback Reranking to select which tokens to retain.
Critical strengths
-
The paper addresses an important limitation of KV caching: deleting an earlier context span does not remove its influence on the contextualized representations of subsequent tokens. Unlike approaches that rely on recompilation of correction patches or insertion of compensating KV states, KVFixer attempts to directly predict the changes required in retained KV states from the original execution. The distinction between representing a correction using paired executions and predicting that correction at inference time is well motivated and constitutes a meaningful contribution.
-
The paper provides a useful separation between representation error, coordinate-prediction error, and downstream output fidelity. The paired comparisons between predicted corrections and exact low-rank projections are particularly informative: although prediction achieves lower mean KL, exact projection performs better on most individual text requests. The inclusion of source-level uncertainty estimates, external evaluation panels, and explicit limitations on exact-state replacement controls adds credibility to the empirical analysis. These observations provide useful insights into the relationship between KV reconstruction and functional behavior.
Critical weaknesses
-
The main comparisons mix differences in scoring and budget allocation. RECAP-KV’s allocation alone raises Expected Attention’s score from 23.0 to 44.6, while matched-budget ranking comparisons remain limited to one model and retention setting. The estimator’s independent contribution remains unclear.
-
The serving benefits remain incompletely validated. RECAP-KV takes 22.7 seconds versus 9.8 seconds for ordinary prefill on one 128K configuration. The vLLM prototype excludes online scoring and end-to-end compression timing, leaving the cost-benefit tradeoff under cache reuse unclear.
Questions for authors
-
Could you compare RECAP-KV, KVzip, and Expected Attention under identical per-head budgets across more models and both retention ratios?
-
Could you report end-to-end latency, throughput, and GPU memory usage, including scoring and compaction, as the number of queries reusing a context increases?
-
How stable are the reference curves across context domains, lengths, and architectures, particularly at aggressive retention ratios?
5 comments
Response to Reviewer w13k (Weakness #2 / Question #2 continued, Question #3)
Weakness #2 / Question #2 (continued)
Throughput and latency as reuse grows
As more queries reuse a context, RECAP-KV's throughput rises from 0.13 to 1.01 requests/s and its mean time to first token falls from 240 s to 33 s, while FullKV stays at 0.24 requests/s and above 230 s; RECAP-KV's accuracy stays within 1.0 to 2.8 points of FullKV's at every reuse count. The reason is what each method pays per query. The 64 uncompressed caches take 7.0 GiB each and do not fit in the 218 GiB KV pool, which holds 31, so FullKV recomputes the prefill for every query, about 4 s each in this benchmark (vLLM's prefill). RECAP-KV pays for scoring and compaction once per context: the 22.7 s of prefill and scoring, plus reference mapping and reassignment, compaction and handing the cache to vLLM, about 30 s in this benchmark. Each later query then costs about 0.1 s. The two break even between 7 and 8 queries per context, and the advantage grows with every query after that: RECAP-KV finishes 2.2× sooner at 16 queries and 4.2× sooner at 32.
GPU memory
GPU memory does not grow with the number of queries. It is set by the cache that each context keeps for as long as it is reused, which is 2.1 GiB with RECAP-KV instead of 7.0 GiB, so the same pool holds 104 contexts instead of 31. Scoring and compaction run in a builder process next to the engine and, measured on their own, peak at 22.5 GiB, about what an ordinary prefill of the same context needs (23.2 GiB).
→ Revised paper: Section 4.4, Table 4 (new); Appendix F (new).
Question #3
Refitting the curves
A reference curve stores, for one layer and head, the typical score at each relative rank. It carries no information about tokens, positions or tasks, and it only decides how the budget is divided among heads; which pairs are kept is decided for every context from that context's own scores. This is why a handful of unlabeled texts is enough to fit it. The first table measures how much the result depends on how the curves are fitted, along each factor the reviewer lists: the curves of Table 1 are replaced by refitted ones and evaluated on the 12-task benchmark at the two budgets of Table 1 and at 10% retention. The first rows of the table give the score with the curves of Table 1, and every other row gives the change relative to it.
| 30% kept | 15% kept | 10% kept | ||
|---|---|---|---|---|
| Reference | Qwen2.5-7B | 62.8 | 54.4 | 42.8 |
| (curves of Table 1) | Llama-3.1-8B | 58.8 | 51.9 | 46.3 |
| Qwen3-30B-A3B | 73.0 | 68.0 | 54.0 |
Change relative to the Reference row of the same model, when the curves are refitted on:
| 30% kept | 15% kept | 10% kept | ||
|---|---|---|---|---|
| Domain | Code only | +0.1 [−0.6, +0.8] | +0.9 [−0.2, +1.9] | +1.5 [+0.2, +2.8] |
| (Qwen2.5-7B) | Mathematics only | +0.2 [−0.5, +1.0] | +0.6 [−0.4, +1.6] | +0.1 [−1.3, +1.3] |
| Translated QA only | −0.1 [−0.8, +0.7] | +1.9 [+0.8, +2.9] | +2.6 [+1.4, +3.7] | |
| Length | 450-token texts | +0.7 [−0.2, +1.5] | +0.7 [−0.6, +1.9] | +0.1 [−1.3, +1.4] |
| (Qwen2.5-7B) | 1,800-token texts | −0.2 [−0.8, +0.4] | +0.1 [−0.7, +0.8] | +0.4 [−0.4, +1.3] |
| 7,200-token texts | −0.5 [−1.1, +0.1] | +0.0 [−1.0, +1.0] | +0.3 [−0.8, +1.4] | |
| 16,384-token texts | −0.6 [−1.4, +0.2] | −1.3 [−2.2, −0.5] | −0.5 [−1.7, +0.7] | |
| Architecture | Qwen2.5-7B | −0.2 [−0.8, +0.3] | +0.9 [−0.2, +1.9] | +0.6 [−0.5, +1.5] |
| (three prose texts) | Llama-3.1-8B | −0.7 [−1.4, +0.1] | +0.4 [−0.7, +1.6] | −1.6 [−2.9, −0.2] |
| Qwen3-30B-A3B | −0.0 [−0.5, +0.4] | −0.5 [−1.3, +0.3] | +1.4 [−0.0, +2.8] |
Reference rows: 12-task mean score (0–100) of RECAP-KV with the reference curves of Table 1. All other rows: RECAP-KV with refitted curves minus the Reference row of the same model (points, with paired 95% intervals), on the same 873 contexts and at the same budget. Positive: the refitted curves score higher. Domain and Length rows: Qwen2.5-7B. Length: six other reference texts (three prose and three key–value texts), each read up to the stated number of tokens. The reference texts of Table 1 are at most 1,800 tokens long. Architecture: the curves of each of three models refitted on the same three prose texts; Qwen3-30B-A3B is a mixture-of-experts model.
Overall result and domains
At the two budgets of Table 1, none of these changes moves the 12-task mean by 2 points or more. Across domains, curves fitted only on code, only on mathematics or only on translated QA change it by −0.1 to +0.2 points at 30% retention and by +0.6 to +1.9 at 15%.
Response to Reviewer w13k (Question #3 continued)
Question #3 (continued)
Context length
Context length enters in two places. The first is the length of the reference texts. The Length rows refit the curves on six other texts, read up to 450, 1,800, 7,200 or 16,384 tokens each, a range of 36 to one; the 12-task mean changes by −1.3 to +0.7 points at the budgets of Table 1, and only one of these eight changes is distinguishable from zero. Longer reference texts thus bring no gain over the short texts of Table 1. The second is the length of the context to be compressed. If long contexts needed curves fitted on long texts, longer reference texts would raise the score on them, and the second table tests this by splitting the Length rows by the length of the evaluated contexts.
| Length of the evaluated contexts (tokens) | Under 8K | 18K to 27K | 65K to 150K |
|---|---|---|---|
| Reference (curves of Table 1) | 76.8 | 49.4 | 45.6 |
Change relative to the Reference row, when the curves are refitted on:
| Length of the evaluated contexts (tokens) | Under 8K | 18K to 27K | 65K to 150K |
|---|---|---|---|
| 450-token texts | −0.9 [−4.5, +2.7] | +3.3 [+1.9, +4.8] | +0.1 [−1.4, +1.8] |
| 1,800-token texts | +0.2 [−2.3, +2.5] | −0.8 [−1.7, +0.1] | +0.4 [−0.3, +1.2] |
| 7,200-token texts | +1.4 [−1.8, +4.5] | −0.8 [−2.0, +0.4] | −0.3 [−1.2, +0.6] |
| 16,384-token texts | −2.8 [−5.5, −0.1] | −1.5 [−2.7, −0.3] | −0.5 [−1.4, +0.2] |
Qwen2.5-7B, 15% retention: the Length rows of the first table split by the length of the evaluated contexts. Reference row: score of RECAP-KV with the curves of Table 1. Other rows: RECAP-KV with the refitted curves minus the Reference row on the same contexts (points, with paired 95% intervals). Scores are means over the tasks whose contexts have that length (233, 254 and 386 contexts).
Longer reference texts do not raise the score on the long contexts: on the contexts of 65K to 150K tokens the score changes by −0.5 to +0.4 points across the four reference lengths, and no change is distinguishable from zero. One set of curves fitted on texts of at most 1,800 tokens is therefore used at every context length; the curves are defined over relative rank and need no refit, from under 100 tokens to 150K tokens within Table 1. Table B3 applies the same curves to key–value retrieval and prefix–suffix copying at up to 169K tokens, where RECAP-KV stays above KVzip, which uses no reference curves and derives each head's budget from the context itself, in 10 of the 12 cells, including every length at 15% retention.
Architectures
Across architectures the curves are not transferred: they are fitted separately for each model, which needs no labels and takes about one second on Qwen2.5-7B. The six models of Table 1, which include mixture-of-experts and hybrid models, all use curves obtained in this way. The Architecture rows refit the curves of three of them, each on the same three prose texts, and the picture is the same on each: the 12-task mean changes by −0.7 to +0.9 points at the budgets of Table 1, and no change is distinguishable from zero.
Aggressive retention
At 10% retention, below the budgets of Table 1, the changes of the first table move the 12-task mean by −1.6 to +2.6 points. The dependence on the fit therefore grows as the budget shrinks, from at most 0.7 points at 30% to 1.9 at 15% and 2.6 at 10%, but it stays below 3 points for every change in the first table.
Below 10% we use a more limited experiment than the 12-task benchmark: the needle test of Figure 3(c), on one model, Qwen3-30B-A3B, over five context lengths from 8K to 128K. The third table gives this test with the curves of Table 1 and with curves fitted on the same three prose texts as in the first table.
| KV retained | 30% | 15% | 10% | 7% | 5% |
|---|---|---|---|---|---|
| RECAP-KV, curves of Table 1 | 220 | 220 | 219 | 172 | 52 |
| RECAP-KV, three prose texts | 220 | 220 | 218 | 181 | 87 |
| KVzip (no reference curves) | 220 | 219 | 218 | 104 | 19 |
Qwen3-30B-A3B-Instruct-2507, needle test of Figure 3(c): needles found out of 220 (44 at each of the context lengths 8K, 16K, 32K, 64K and 128K; the uncompressed cache finds all 220). The two RECAP-KV rows differ only in the reference curves, those of Table 1 and curves fitted on three prose texts; the second row is the series plotted in Figure 3(c). The caption of Figure 3 in the revised paper now states which reference curves panels (a) and (c) use. Bold: RECAP-KV where it finds clearly more needles than KVzip.
Down to 10% retention the two sets of curves agree to within one needle, and so does KVzip. At 7% they differ by 9 needles and at 5% by 35, so below 10% the choice of curves matters. The comparison with KVzip does not change: with either set RECAP-KV finds 68 to 77 more needles at 7% and 33 to 68 more at 5%.
→ Revised paper: Appendix B.3, Tables B8, B9 and B10 (new).
Response to Reviewer w13k (Weakness #1 / Question #1, Weakness #2 / Question #2)
We thank the reviewer for the careful questions on how allocation and scoring contribute, on serving cost under reuse, and on the stability of the reference curves. We answer each below.
Weakness #1 / Question #1
Same allocation for all three methods
RECAP-KV has two components, and the comparison the reviewer asks for separates them: its reference-curve allocation decides how many KV pairs each head keeps, and its estimator decides which pairs. All three methods are given the same allocation, RECAP-KV's, so the only thing that differs is which pairs each method keeps.
12-task mean score (0–100) over 873 contexts per model. The first score column is EA with its own equal per-head budgets (Table 1). The next three columns use RECAP-KV's allocation, that is, in each layer and head all three keep the number of KV pairs that RECAP-KV keeps there, and differ only in the ranking. EA, like RECAP-KV, ranks during a single prefill; KVzip needs a second pass over the context. EA: Expected Attention. Bold: RECAP-KV where it has the best score of the row, and its lead over EA's ranking, positive in all six settings.
The estimator's own contribution
The table shows two things about the estimator. First, compared with EA, the other ranking that is computed during a single prefill, RECAP-KV is better in all six settings, by 1.2 to 11.7 points. This does not depend on our allocation: when every head simply receives an equal budget, RECAP-KV's ranking still leads EA's by 5.1 to 12.2 points on Qwen2.5 and Llama-3.1 (the equal per-head row of Table 2B against EA in Table 1). Second, compared with KVzip, which needs a second pass over the context to compute its ranking, RECAP-KV is on par: higher on Llama-3.1, the same within noise on Qwen3.8-27B (both intervals include zero), and 1 to 2 points lower on Qwen2.5.
Both components contribute
Both components therefore contribute, each on its own. The allocation is what the reviewer's example measures, and the first two score columns show it: given RECAP-KV's per-head budgets, EA improves in all six settings, by 6.5 to 30.3 points (from 23.0 to 44.6 in the reviewer's example). The estimator accounts for what remains: with that very same allocation, EA is still 1.2 to 11.7 points below RECAP-KV, whose single-prefill ranking is on par with the two-pass ranking of KVzip.
→ Revised paper: Section 4.3, Table 3 (extended to three models).
Weakness #2 / Question #2
End-to-end benchmark
In an end-to-end benchmark with scoring and compaction included, 64 users each bring a different 128K-token document, all start at the same time, and each context is reused by 4 to 32 queries (Qwen2.5-7B-Instruct-1M, vLLM 0.27.1, one B300 GPU). RECAP-KV scores and compacts a context on the same GPU when its first query arrives, and that time counts in the latency and throughput below. FullKV is stock vLLM with automatic prefix caching.
64 users, 128K-token documents, 30% retention; answers are up to 32 tokens. TTFT: time to first token, averaged over all requests, including each context's first query, which waits for scoring and compaction. Accuracy: exact match of the answer phrase over all requests. The better requests/s and TTFT of each row are in bold.