acceptodds
Under review as a conference paper at ICLR 2027

RECAP-KV: Single-Prefill Approximation of Reconstruction Importance for Reusable KV Cache Eviction

Abstract

Key–value (KV) caching avoids recomputing the attention keys and values of processed tokens during generation. It also lets a long document, once processed, serve many subsequent queries. However, the cache grows with context length, limiting batch size and the number of caches that can remain resident in GPU memory. KV eviction addresses this by retaining only a subset of the cached KV pairs. Yet identifying KV pairs that preserve answer quality for unknown future queries is challenging, and reliable importance estimation is expensive for long contexts. We present RECAP-KV, a query-agnostic eviction method that reduces this cost while maintaining competitive answer quality. It applies the model's existing Q/K projections and normalization to attention-input states captured during prefill and uses rotary positional embeddings at virtual positions to approximate the attention that cached keys would receive during context reconstruction, without an additional transformer pass or a learned importance predictor. RECAP-KV allocates the cache budget across layers, heads, and context blocks, and its Lookback Reranking refines token selection without changing the number of KV pairs retained in each block. On 12 tasks and 6 models from the Qwen, Llama, and GPT-OSS families, RECAP-KV achieves the best or near-best mean task scores among query-agnostic methods at 15% and 30% KV retention, while its prefill-and-scoring time at 128K tokens is 37–46% lower than that of the strongest baseline under matched settings on B300 GPUs.

Primary area:
foundation or frontier models, including LLMs
Keywords:
KV Cache Compression, Query-Agnostic Eviction, Long-Context Inference, KV Cache Reuse, Context Reconstruction

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_w13kOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes RECAP-KV, a query-agnostic KV cache eviction method. It approximates KVzip’s reconstruction-based importance scores without an additional Transformer forward pass or a trained predictor. RECAP-KV constructs virtual queries from cached hidden states. It uses reference curves to allocate the retention budget, then applies Lookback Reranking to select which tokens to retain.

Critical strengths
  1. The paper addresses an important limitation of KV caching: deleting an earlier context span does not remove its influence on the contextualized representations of subsequent tokens. Unlike approaches that rely on recompilation of correction patches or insertion of compensating KV states, KVFixer attempts to directly predict the changes required in retained KV states from the original execution. The distinction between representing a correction using paired executions and predicting that correction at inference time is well motivated and constitutes a meaningful contribution.

  2. The paper provides a useful separation between representation error, coordinate-prediction error, and downstream output fidelity. The paired comparisons between predicted corrections and exact low-rank projections are particularly informative: although prediction achieves lower mean KL, exact projection performs better on most individual text requests. The inclusion of source-level uncertainty estimates, external evaluation panels, and explicit limitations on exact-state replacement controls adds credibility to the empirical analysis. These observations provide useful insights into the relationship between KV reconstruction and functional behavior.

Critical weaknesses
  1. The main comparisons mix differences in scoring and budget allocation. RECAP-KV’s allocation alone raises Expected Attention’s score from 23.0 to 44.6, while matched-budget ranking comparisons remain limited to one model and retention setting. The estimator’s independent contribution remains unclear.

  2. The serving benefits remain incompletely validated. RECAP-KV takes 22.7 seconds versus 9.8 seconds for ordinary prefill on one 128K configuration. The vLLM prototype excludes online scoring and end-to-end compression timing, leaving the cost-benefit tradeoff under cache reuse unclear.

Questions for authors
  1. Could you compare RECAP-KV, KVzip, and Expected Attention under identical per-head budgets across more models and both retention ratios?

  2. Could you report end-to-end latency, throughput, and GPU memory usage, including scoring and compaction, as the number of queries reusing a context increases?

  3. How stable are the reference curves across context domains, lengths, and architectures, particularly at aggressive retention ratios?

5 comments
AuthorsOpenReview ↗

Response to Reviewer w13k (Weakness #1 / Question #1, Weakness #2 / Question #2)

We thank the reviewer for the careful questions on how allocation and scoring contribute, on serving cost under reuse, and on the stability of the reference curves. We answer each below.


Weakness #1 / Question #1

Same allocation for all three methods

RECAP-KV has two components, and the comparison the reviewer asks for separates them: its reference-curve allocation decides how many KV pairs each head keeps, and its estimator decides which pairs. All three methods are given the same allocation, RECAP-KV's, so the only thing that differs is which pairs each method keeps.

ModelKV keptEA, own alloc.EA rank.KVzip rank.RECAP-KVRECAP-KV − EARECAP-KV − KVzip
Qwen2.5-7B30%31.361.663.862.8+1.17−0.99
15%23.044.656.154.4+9.79−1.75
Llama-3.1-8B30%31.455.856.358.8+2.99+2.50
15%23.547.650.451.9+4.36+1.50
Qwen3.8-27B30%44.265.376.077.0+11.66+0.98
15%38.044.550.651.3+6.85+0.75

12-task mean score (0–100) over 873 contexts per model. The first score column is EA with its own equal per-head budgets (Table 1). The next three columns use RECAP-KV's allocation, that is, in each layer and head all three keep the number of KV pairs that RECAP-KV keeps there, and differ only in the ranking. EA, like RECAP-KV, ranks during a single prefill; KVzip needs a second pass over the context. EA: Expected Attention. Bold: RECAP-KV where it has the best score of the row, and its lead over EA's ranking, positive in all six settings.

The estimator's own contribution

The table shows two things about the estimator. First, compared with EA, the other ranking that is computed during a single prefill, RECAP-KV is better in all six settings, by 1.2 to 11.7 points. This does not depend on our allocation: when every head simply receives an equal budget, RECAP-KV's ranking still leads EA's by 5.1 to 12.2 points on Qwen2.5 and Llama-3.1 (the equal per-head row of Table 2B against EA in Table 1). Second, compared with KVzip, which needs a second pass over the context to compute its ranking, RECAP-KV is on par: higher on Llama-3.1, the same within noise on Qwen3.8-27B (both intervals include zero), and 1 to 2 points lower on Qwen2.5.

Both components contribute

Both components therefore contribute, each on its own. The allocation is what the reviewer's example measures, and the first two score columns show it: given RECAP-KV's per-head budgets, EA improves in all six settings, by 6.5 to 30.3 points (from 23.0 to 44.6 in the reviewer's example). The estimator accounts for what remains: with that very same allocation, EA is still 1.2 to 11.7 points below RECAP-KV, whose single-prefill ranking is on par with the two-pass ranking of KVzip.

→ Revised paper: Section 4.3, Table 3 (extended to three models).


Weakness #2 / Question #2

End-to-end benchmark

In an end-to-end benchmark with scoring and compaction included, 64 users each bring a different 128K-token document, all start at the same time, and each context is reused by 4 to 32 queries (Qwen2.5-7B-Instruct-1M, vLLM 0.27.1, one B300 GPU). RECAP-KV scores and compacts a context on the same GPU when its first query arrives, and that time counts in the latency and throughput below. FullKV is stock vLLM with automatic prefix caching.

Queries reusing each contextFullKV req/s ↑FullKV TTFT (s) ↓FullKV acc. ↑RECAP-KV req/s ↑RECAP-KV TTFT (s) ↓RECAP-KV acc. ↑
40.24233.80.9380.13239.70.910
60.24239.40.9270.20160.60.904
80.24250.10.9200.27121.70.908
160.24258.20.9260.5262.90.916
320.24262.40.9211.0132.60.906

64 users, 128K-token documents, 30% retention; answers are up to 32 tokens. TTFT: time to first token, averaged over all requests, including each context's first query, which waits for scoring and compaction. Accuracy: exact match of the answer phrase over all requests. The better requests/s and TTFT of each row are in bold.

AuthorsOpenReview ↗

Response to Reviewer w13k (Weakness #2 / Question #2 continued, Question #3)

Weakness #2 / Question #2 (continued)

Throughput and latency as reuse grows

As more queries reuse a context, RECAP-KV's throughput rises from 0.13 to 1.01 requests/s and its mean time to first token falls from 240 s to 33 s, while FullKV stays at 0.24 requests/s and above 230 s; RECAP-KV's accuracy stays within 1.0 to 2.8 points of FullKV's at every reuse count. The reason is what each method pays per query. The 64 uncompressed caches take 7.0 GiB each and do not fit in the 218 GiB KV pool, which holds 31, so FullKV recomputes the prefill for every query, about 4 s each in this benchmark (vLLM's prefill). RECAP-KV pays for scoring and compaction once per context: the 22.7 s of prefill and scoring, plus reference mapping and reassignment, compaction and handing the cache to vLLM, about 30 s in this benchmark. Each later query then costs about 0.1 s. The two break even between 7 and 8 queries per context, and the advantage grows with every query after that: RECAP-KV finishes 2.2× sooner at 16 queries and 4.2× sooner at 32.

GPU memory

GPU memory does not grow with the number of queries. It is set by the cache that each context keeps for as long as it is reused, which is 2.1 GiB with RECAP-KV instead of 7.0 GiB, so the same pool holds 104 contexts instead of 31. Scoring and compaction run in a builder process next to the engine and, measured on their own, peak at 22.5 GiB, about what an ordinary prefill of the same context needs (23.2 GiB).

→ Revised paper: Section 4.4, Table 4 (new); Appendix F (new).


Question #3

Refitting the curves

A reference curve stores, for one layer and head, the typical score at each relative rank. It carries no information about tokens, positions or tasks, and it only decides how the budget is divided among heads; which pairs are kept is decided for every context from that context's own scores. This is why a handful of unlabeled texts is enough to fit it. The first table measures how much the result depends on how the curves are fitted, along each factor the reviewer lists: the curves of Table 1 are replaced by refitted ones and evaluated on the 12-task benchmark at the two budgets of Table 1 and at 10% retention. The first rows of the table give the score with the curves of Table 1, and every other row gives the change relative to it.

30% kept15% kept10% kept
ReferenceQwen2.5-7B62.854.442.8
(curves of Table 1)Llama-3.1-8B58.851.946.3
Qwen3-30B-A3B73.068.054.0

Change relative to the Reference row of the same model, when the curves are refitted on:

30% kept15% kept10% kept
DomainCode only+0.1 [−0.6, +0.8]+0.9 [−0.2, +1.9]+1.5 [+0.2, +2.8]
(Qwen2.5-7B)Mathematics only+0.2 [−0.5, +1.0]+0.6 [−0.4, +1.6]+0.1 [−1.3, +1.3]
Translated QA only−0.1 [−0.8, +0.7]+1.9 [+0.8, +2.9]+2.6 [+1.4, +3.7]
Length450-token texts+0.7 [−0.2, +1.5]+0.7 [−0.6, +1.9]+0.1 [−1.3, +1.4]
(Qwen2.5-7B)1,800-token texts−0.2 [−0.8, +0.4]+0.1 [−0.7, +0.8]+0.4 [−0.4, +1.3]
7,200-token texts−0.5 [−1.1, +0.1]+0.0 [−1.0, +1.0]+0.3 [−0.8, +1.4]
16,384-token texts−0.6 [−1.4, +0.2]−1.3 [−2.2, −0.5]−0.5 [−1.7, +0.7]
ArchitectureQwen2.5-7B−0.2 [−0.8, +0.3]+0.9 [−0.2, +1.9]+0.6 [−0.5, +1.5]
(three prose texts)Llama-3.1-8B−0.7 [−1.4, +0.1]+0.4 [−0.7, +1.6]−1.6 [−2.9, −0.2]
Qwen3-30B-A3B−0.0 [−0.5, +0.4]−0.5 [−1.3, +0.3]+1.4 [−0.0, +2.8]

Reference rows: 12-task mean score (0–100) of RECAP-KV with the reference curves of Table 1. All other rows: RECAP-KV with refitted curves minus the Reference row of the same model (points, with paired 95% intervals), on the same 873 contexts and at the same budget. Positive: the refitted curves score higher. Domain and Length rows: Qwen2.5-7B. Length: six other reference texts (three prose and three key–value texts), each read up to the stated number of tokens. The reference texts of Table 1 are at most 1,800 tokens long. Architecture: the curves of each of three models refitted on the same three prose texts; Qwen3-30B-A3B is a mixture-of-experts model.

Overall result and domains

At the two budgets of Table 1, none of these changes moves the 12-task mean by 2 points or more. Across domains, curves fitted only on code, only on mathematics or only on translated QA change it by −0.1 to +0.2 points at 30% retention and by +0.6 to +1.9 at 15%.

AuthorsOpenReview ↗

Response to Reviewer w13k (Question #3 continued)

Question #3 (continued)

Context length

Context length enters in two places. The first is the length of the reference texts. The Length rows refit the curves on six other texts, read up to 450, 1,800, 7,200 or 16,384 tokens each, a range of 36 to one; the 12-task mean changes by −1.3 to +0.7 points at the budgets of Table 1, and only one of these eight changes is distinguishable from zero. Longer reference texts thus bring no gain over the short texts of Table 1. The second is the length of the context to be compressed. If long contexts needed curves fitted on long texts, longer reference texts would raise the score on them, and the second table tests this by splitting the Length rows by the length of the evaluated contexts.

Length of the evaluated contexts (tokens)Under 8K18K to 27K65K to 150K
Reference (curves of Table 1)76.849.445.6

Change relative to the Reference row, when the curves are refitted on:

Length of the evaluated contexts (tokens)Under 8K18K to 27K65K to 150K
450-token texts−0.9 [−4.5, +2.7]+3.3 [+1.9, +4.8]+0.1 [−1.4, +1.8]
1,800-token texts+0.2 [−2.3, +2.5]−0.8 [−1.7, +0.1]+0.4 [−0.3, +1.2]
7,200-token texts+1.4 [−1.8, +4.5]−0.8 [−2.0, +0.4]−0.3 [−1.2, +0.6]
16,384-token texts−2.8 [−5.5, −0.1]−1.5 [−2.7, −0.3]−0.5 [−1.4, +0.2]

Qwen2.5-7B, 15% retention: the Length rows of the first table split by the length of the evaluated contexts. Reference row: score of RECAP-KV with the curves of Table 1. Other rows: RECAP-KV with the refitted curves minus the Reference row on the same contexts (points, with paired 95% intervals). Scores are means over the tasks whose contexts have that length (233, 254 and 386 contexts).

Longer reference texts do not raise the score on the long contexts: on the contexts of 65K to 150K tokens the score changes by −0.5 to +0.4 points across the four reference lengths, and no change is distinguishable from zero. One set of curves fitted on texts of at most 1,800 tokens is therefore used at every context length; the curves are defined over relative rank and need no refit, from under 100 tokens to 150K tokens within Table 1. Table B3 applies the same curves to key–value retrieval and prefix–suffix copying at up to 169K tokens, where RECAP-KV stays above KVzip, which uses no reference curves and derives each head's budget from the context itself, in 10 of the 12 cells, including every length at 15% retention.

Architectures

Across architectures the curves are not transferred: they are fitted separately for each model, which needs no labels and takes about one second on Qwen2.5-7B. The six models of Table 1, which include mixture-of-experts and hybrid models, all use curves obtained in this way. The Architecture rows refit the curves of three of them, each on the same three prose texts, and the picture is the same on each: the 12-task mean changes by −0.7 to +0.9 points at the budgets of Table 1, and no change is distinguishable from zero.

Aggressive retention

At 10% retention, below the budgets of Table 1, the changes of the first table move the 12-task mean by −1.6 to +2.6 points. The dependence on the fit therefore grows as the budget shrinks, from at most 0.7 points at 30% to 1.9 at 15% and 2.6 at 10%, but it stays below 3 points for every change in the first table.

Below 10% we use a more limited experiment than the 12-task benchmark: the needle test of Figure 3(c), on one model, Qwen3-30B-A3B, over five context lengths from 8K to 128K. The third table gives this test with the curves of Table 1 and with curves fitted on the same three prose texts as in the first table.

KV retained30%15%10%7%5%
RECAP-KV, curves of Table 122022021917252
RECAP-KV, three prose texts22022021818187
KVzip (no reference curves)22021921810419

Qwen3-30B-A3B-Instruct-2507, needle test of Figure 3(c): needles found out of 220 (44 at each of the context lengths 8K, 16K, 32K, 64K and 128K; the uncompressed cache finds all 220). The two RECAP-KV rows differ only in the reference curves, those of Table 1 and curves fitted on three prose texts; the second row is the series plotted in Figure 3(c). The caption of Figure 3 in the revised paper now states which reference curves panels (a) and (c) use. Bold: RECAP-KV where it finds clearly more needles than KVzip.

Down to 10% retention the two sets of curves agree to within one needle, and so does KVzip. At 7% they differ by 9 needles and at 5% by 35, so below 10% the choice of curves matters. The comparison with KVzip does not change: with either set RECAP-KV finds 68 to 77 more needles at 7% and 33 to 68 more at 5%.

→ Revised paper: Appendix B.3, Tables B8, B9 and B10 (new).

Reviewer_aEXNOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

RECAP-KV is a query-agnostic KV cache eviction method. It compresses a long context's KV cache for reuse across future queries. It uses temporary virtual Q/K vectors during the initial prefill to estimate which original KV pairs are important, retains the highest-scoring pairs, and discards the rest. It achieves quality close to reconstruction-based eviction while avoiding a second full transformer pass.

Critical strengths
  1. It uses model's existing QK projections and RoPE rather than requiring a separately trained, model-specific importance predictor.

  2. It evaluates 12 tasks and six models including dense, MoE, and hybrid-attention architectures at multiple retention levels and context lengths.

  3. It provides a vLLM prototype demonstrating that nonuniform per-layer and per-head retention can be represented in paged KV storage.

  4. It includes component ablations, task-level results, score-agreement analysis, timing and memory measurements, and confidence intervals for key comparisons.

Critical weaknesses
  1. There is no complete end-to-end serving evaluation. The paper does not demonstrate throughput or latency gains under realistic concurrent, multi-query workloads.

  2. The cache-construction overhead remains. RECAP-KV performs additional QK projections and attention scoring. At 128K on Qwen2.5-7B, it takes 22.7 seconds vs 9.8 seconds for ordinary prefill

  3. The method assumes a stable share-prefix context with consistent logical positions.

  4. The benchmark suite is broad, but it remains unclear how well reconstruction importance transfer to unseen query distributions.

Questions for authors
  1. Can the authors report end-to-end latency, throughput, and peak memory for cache construction

  2. How many subsequent queries are required to compensate RECAP-KV’s additional scoring and compaction cost

  3. Can the authors compare more systematically with Fast KVzip and KVzap under matched models, budgets, training data, and allocation policies?

  4. How sensitive are the reference curves to domain shift, model quantization, fine-tuning, LoRA adapters, and changes in context length?

  5. Does the method support contexts appearing at different logical positions, or is reuse restricted to an identical shared prefix? Please clarify this operational limitation and its implications.

7 comments
AuthorsOpenReview ↗

Response to Reviewer aEXN (Weakness #1, Weakness #2)

We thank the reviewer for the constructive review.


Weakness #1

End-to-end serving benchmark

In an end-to-end serving benchmark with cache construction included, D users, each with a different 128K-token document, start at the same time, and each asks 16 queries one after another, on one B300 GPU (Qwen2.5-7B-Instruct-1M, vLLM 0.27.1). FullKV is stock vLLM with automatic prefix caching. RECAP-KV and KVzip build each document's compressed cache (30% retention) on the same GPU when its first query arrives, and the construction time is included in the latency and throughput numbers below.

Users × queriesMethodRequests/s ↑Mean TTFT (s) ↓Total time (s) ↓Accuracy ↑
32 × 16FullKV0.28107.71,814.90.906
KVzip0.3646.31,422.60.891
RECAP-KV0.5231.9978.70.883
64 × 16FullKV0.24258.24,287.20.926
KVzip0.3690.52,839.00.921
RECAP-KV0.5262.91,969.30.916
128 × 16FullKV0.24515.38,530.80.931
KVzip0.36178.45,657.80.912
RECAP-KV0.53122.53,876.80.915

End-to-end serving with 128K-token documents, cache construction included. TTFT: time to first token, averaged over all requests, including each document's first query, which waits for cache construction. Answers are up to 32 tokens; accuracy is the exact match of the answer phrase over all requests. The best requests/s, TTFT and total time of each block are in bold.

Throughput and latency

RECAP-KV serves 1.9–2.2× as many requests per second as FullKV, and its mean time to first token is 3.4–4.2× lower with the construction wait included. RECAP-KV is also 1.4–1.5× faster than KVzip at the same retention.

Where the gain comes from

These values follow from how often each method has to process a document. A 128K-token document takes 7.0 GiB of KV cache, so the 218 GiB pool holds about 31 documents. With 32 or more users, FullKV evicts cached prefixes and recomputes the prefill for every query. Each request then costs about 4 s of GPU time (vLLM's prefill), which gives about 0.25 requests/s, and each query also waits for the other users' prefills, so its time to first token grows in proportion to the number of users. RECAP-KV processes each document once, to build its cache (about 30 s; 2.1 GiB at 30% retention), and then starts answering each of that document's 16 queries in 0.4 s (time to first token). Sixteen requests per 30 s gives about 0.5 requests/s, and only the first of the 16 queries waits for construction. KVzip serves queries the same way but needs about 44 s to build each cache, which gives about 0.36 requests/s. The gain over FullKV therefore appears once more documents are in use than fit in GPU memory uncompressed and each document is queried several times; our answer to Question #2 gives the break-even. The smaller cache also brings two gains that do not depend on recomputation: the same pool holds about 104 contexts instead of 31, and decoding is 1.7× faster (Weakness #2).

→ Revised paper: Section 4.4, Table 4 (new); Appendix F (new).


Weakness #2

What the extra time buys

The extra 12.9 s is paid once per context, and it buys a cache that is 3.3× smaller for every later query. At 30% retention, a 128K-token context keeps 2.1 GiB of KV cache instead of 7.0 GiB. The 218 GiB KV pool in Weakness #1 therefore has room for about 104 such contexts instead of 31. Decoding is also 1.7× faster, because each step reads less of the cache: 16 concurrent 128K-token requests decode at 1,058 tokens/s instead of 611 in unmodified vLLM (597 in our vLLM prototype with the full cache), or 15.1 ms per decoding step instead of 26.2 ms. End to end, with construction included, RECAP-KV serves 1.9–2.2× as many requests per second as FullKV (Weakness #1).

Compared with reconstruction

RECAP-KV also keeps this cost to less than half of what reconstruction costs. KVzip, the reconstruction-based method that RECAP-KV approximates, runs a second pass over the context and adds 28.2 s to the 9.8 s prefill. RECAP-KV scores during the prefill itself and adds 12.9 s. On all six models in Table 1C, RECAP-KV's added time is 16–49% of KVzip's, while its mean score matches or exceeds KVzip's on all six models at 30% retention. Our answer to Question #2 gives the number of queries after which the cost is recovered.

→ Revised paper: Appendix F, Table F3 (new).

AuthorsOpenReview ↗

Response to Reviewer aEXN (Weakness #3 / Question #5, Weakness #4)

Weakness #3 / Question #5

Reuse at a different position

Reuse is not restricted to an identical prefix: a cache compressed by RECAP-KV can be moved to a different position in the same way as an uncompressed cache. Each cached key carries its token position through the rotary position encoding, so when the document moves we only rotate the retained keys by the position offset, and the document is not processed again. To test this, we built each document's cache once at its original position and answered the same queries twice: once with the document at its original position, and once with the same cache moved behind an unrelated 2K- or 16K-token prefix. If compression made a cache harder to move, the score of the compressed cache would change more under this move than the score of the uncompressed cache. We find no such effect. The table gives the score change of RECAP-KV under the move minus the score change of the uncompressed cache under the same move: the differences lie between −1.25 and +1.89 points, and all eight 95% intervals include zero.

ModelRetention2K-token prefix16K-token prefix
Qwen2.5-7B-1M30%+1.00 [−0.51, +2.52]−0.52 [−2.42, +1.40]
Qwen2.5-7B-1M15%+0.46 [−1.26, +2.21]+1.08 [−0.95, +3.15]
Llama-3.1-8B30%−0.27 [−2.64, +2.14]−0.90 [−3.92, +2.07]
Llama-3.1-8B15%+1.89 [−0.72, +4.54]−1.25 [−4.43, +1.84]

Score change of RECAP-KV under the move minus the score change of the uncompressed cache under the same move. Points, with paired 95% bootstrap intervals. Qwen2.5-7B-Instruct-1M: 278 contexts, eight SCBench tasks. Llama-3.1-8B-Instruct: 234 contexts, seven tasks (variable tracking does not fit its 128K window with a 16K prefix). Positive: the compressed cache loses less under the move than the uncompressed cache.

Implications

Therefore one compressed cache per document serves prompts in which the document follows different preceding text, and construction is still paid once per document. A moved cache keeps the states computed with the original preceding text, exactly as an uncompressed cache does, so RECAP-KV adds no operational restriction beyond that of moving an uncompressed cache.

→ Revised paper: Appendix B.5, Table B19 (new).


Weakness #4

Queries beyond recall

A score built from reconstruction would be a concern if it served only queries that resemble reconstruction. Tables B1 and B2 report every task of Table 1 separately and show that it also serves queries that ask for more than recalling the text: GSM8K, math finding and variable tracking require computing over the context, and summarization and many-shot learning require generating new text or learning from in-context examples. Averaged over the six models, RECAP-KV at 30% retention is at most 0.5 points below FullKV on each of these five tasks (table below). This is what the criterion predicts. Reproducing a context requires all of its content, whereas a downstream query uses part of it, so a selection that keeps what reconstruction needs also covers queries of other kinds.

TaskQuery typeFullKVKVzipRECAP-KV
GSM8Kcomputing69.868.070.5
Math findingcomputing41.339.341.2
Variable trackingcomputing57.758.757.2
Summarizationgeneration36.636.736.4
Many-shot learningin-context learning54.456.455.5

Score (0–100) ↑, mean over the six models of Table 1 at 30% retention, from the task-level Tables B1 and B2. KVzip computes reconstruction importance exactly with a second pass; RECAP-KV estimates it during the prefill.

No query or label used; RULER

RECAP-KV computes its scores from the context alone and fits its reference curves on unlabeled text, so compressing a context uses no query, task label or answer. RULER tests it on 13 tasks from a benchmark that Table 1 does not use. With the curves and settings of Table 1 unchanged, RECAP-KV scores 92.1 at 30% retention and 88.0 at 15% on its 260 contexts, against 92.8 for FullKV (Table D1, Qwen2.5-7B-Instruct-1M).

→ Revised paper: Appendix B.1, Table B4 (new).


AuthorsOpenReview ↗

Response to Reviewer aEXN (Question #1, Question #2)

Question #1

Construction cost

Building the compressed cache of a 128K-token context takes RECAP-KV 23.81 s (5.5K tokens/s), needs about the same GPU memory as an ordinary prefill (22.5 against 23.2 GiB at peak), and leaves a cache that is 3.3× smaller (2.1 against 7.0 GiB). KVzip needs 38.76 s for the same compression, 1.6× longer. The table gives the full measurements for Qwen2.5-7B-Instruct-1M on one B300 GPU.

Ordinary prefill (no compression)KVzip (30%)RECAP-KV (30%)
Construction
Time, 32K tokens (s)1.033.923.27
Time, 64K tokens (s)3.0111.618.18
Time, 128K tokens (s)9.9738.7623.81
Throughput, 128K tokens (tokens/s)13.1K3.4K5.5K
Peak GPU memory, 128K tokens (GiB)23.223.222.5
After construction
KV cache per 128K-token context (GiB)7.02.12.1
128K-token contexts that fit in a 218 GiB KV pool31104104
Decoding, 16 concurrent 128K-token requests (tokens/s)6111,058†1,058†

Cache construction and its result. Median of 5 runs after 2 warm-ups; 128K = 130,816 tokens. Bold: the faster of the two compressed caches. Each method uses its default prefill piece size (16,000 tokens for the ordinary prefill and KVzip, 8,000 for RECAP-KV). Peak memory includes 14.2 GiB of model weights; with 8,000-token pieces, the ordinary prefill and KVzip peak at 22.1 and 22.3 GiB. Decoding was measured in vLLM 0.27.1 with 512 generated tokens per request, as the mean of 16 rounds for each compressed cache and of 4 rounds without compression. †Common mean of the two compressed caches: their rates cannot be told apart (ratio of the two rates 0.99, 95% interval 0.93 to 1.05).

What the smaller cache changes

A 128K-token context occupies 7.0 GiB of KV cache on this model, half the size of the weights (14.2 GiB), and the cache has to stay in GPU memory for as long as the context is reused. The 218 GiB KV pool of one GPU therefore holds 31 such contexts. A context beyond that number loses its cache and is recomputed on its next query, about 4 s each (vLLM's prefill), and every decoding step reads the whole cache, so decoding slows down as more contexts are decoded together. Keeping 30% of the KV pairs changes all three: the cache shrinks to 2.1 GiB, the pool holds 104 contexts (208 at 15% retention), and 16 concurrent requests decode at 1,058 tokens/s instead of 611, while the 12-task score is 62.8 against 65.7 for the full cache (Table 1A). The construction time in the upper rows is the price of this, paid once per context. Peak memory during construction is set by the prefill itself, when every method holds the full 7.0 GiB cache, so the memory benefit appears only after construction.

Breakdown of the construction time

Of the 23.81 s, prefill and scoring take 22.8 s (the 22.7 s of Table 1C), reference mapping and the reassignment step of Lookback Reranking 1.0 s, and KV selection and compaction 0.03 s. In the serving benchmark of Weakness #1, the same construction takes about 30 s per document, mainly because the cache is also packed for vLLM and transferred to the engine.

→ Revised paper: Appendix F, Table F3 (new).


Question #2

Break-even point

About 7 to 8 queries per context. Once more contexts are in use than fit in GPU memory uncompressed, FullKV recomputes the prefill for every query, about 4 s of GPU time each at 128K tokens in the benchmark of Weakness #1 (vLLM's prefill), while RECAP-KV pays about 30 s of construction once per context and about 0.1 s of GPU time for each query after that. After n queries, a context has therefore cost about 4n s with FullKV and 30 + 0.1n s with RECAP-KV, and the two cross between 7 and 8 queries. The table shows this crossover in the benchmark of Weakness #1.

Measured totals

UsersQueries per user4681632
32FullKV (s)478.5693.7938.21,814.93,669.2
RECAP-KV (s)929.1941.4933.5978.71,022.5
FullKV / RECAP-KV0.520.741.011.853.59
64FullKV (s)1,071.71,573.22,144.04,287.28,575.8
RECAP-KV (s)1,901.71,903.31,921.31,969.32,033.6
FullKV / RECAP-KV0.560.831.122.184.22
128FullKV (s)2,105.13,148.04,186.98,530.816,788.1
RECAP-KV (s)3,784.83,801.93,813.43,876.84,043.9
FullKV / RECAP-KV0.560.831.102.204.15

Total time in seconds for 128K-token documents at 30% retention, construction included, and the ratio of the two totals. Bold: RECAP-KV at least 1.5× faster.

FullKV is faster at 4 and 6 queries per user, the two are close at 8 queries (FullKV / RECAP-KV is 1.01 to 1.12), and RECAP-KV is 1.9–2.2× faster at 16 and 3.6–4.2× faster at 32. KVzip, with about 44 s of construction per context, needs about 10 to 12 queries to break even.

→ Revised paper: Appendix F, Table F2 (new).

Reviewer_KUoEOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes RECAP-KV, a KVzip-style KV cache eviction method. To avoid processing the context twice, it reuses hidden states from the original prefill to estimate token importance. Additionally, it employs a KV budget allocation strategy to minimize the performance degradation caused by cache compression.

Critical strengths
  1. Compared with KVZip, the proposed method uses less time to finish the prefill part while reduing the same amount KV cache.
  2. The paper evaluates multiple model architectures across diverse tasks and compression ratios. The ablation studies provide useful evidence for the effectiveness of the proposed components.
Critical weaknesses

Although the paper includes some ablation studies, a more systematic and incremental ablation analysis would be beneficial. Specifically, the authors should progressively incorporate value-norm weighting, reference-curve-based budget allocation, and Lookback Reranking into the Base scoring method to clearly quantify the contribution of each component. More importantly, the authors should investigate how closely the Base scoring method approaches the performance of KVzip, or alternatively, how well KVzip performs when evaluated under the same budget allocation strategy. This would help clarify whether the observed performance improvements primarily stem from the proposed scoring method or the budget allocation strategy.

Questions for authors

See weakness and

  1. While the Base view is used for budget allocation and the Reranking view refines token selection, why the Reranking scores cannot be directly used for both purposes?
5 comments
AuthorsOpenReview ↗

Response to Reviewer KUoE (Weakness #1)

We thank the reviewer for the constructive review.


Weakness #1

Ablation Study: Adding the components one at a time

The table below starts from the Base score with an equal budget for every head and adds RECAP-KV's three components one at a time, in the order in which the method applies them. Each cell gives the score and, in parentheses, the gain over the row above.

ConfigurationQwen2.5-7B, 30%Qwen2.5-7B, 15%Llama-3.1-8B, 30%Llama-3.1-8B, 15%
Base score, equal budget per head42.631.737.224.2
+ value-norm weighting43.9 (+1.3)31.9 (+0.2)39.3 (+2.1)26.2 (+1.9)
+ reference-curve allocation61.4 (+17.5)53.3 (+21.4)55.8 (+16.5)49.3 (+23.2)
+ Lookback Reranking (= RECAP-KV)62.8 (+1.3)54.4 (+1.0)58.8 (+3.0)51.9 (+2.6)
KVzip (second pass over the context)61.450.456.549.2

12-task mean score (0–100) over 873 contexts per model. Gains are computed from unrounded scores. The KVzip row is from Table 1. Bold: RECAP-KV, above KVzip in all four settings.

What each component adds

Each component fixes one specific way in which a score computed from a single prefill falls short, and the table measures what each fix is worth. First, the Base score ranks KV pairs by attention weight alone, but a head's output is a weighted sum of value vectors, so the effect of a pair also depends on its value norm. Weighting the score by that norm improves the score by up to 2.1 points in the four settings of the table. Second, heads differ in how many KV pairs they need, so an equal budget for every head wastes much of the cache. The reference curves, computed once per model, put all heads on a common scale and give each head its own budget. This is the largest step, 16.5 to 23.2 points. Third, the Base score evaluates a block from the position right after it, whereas reconstruction evaluates it from the end of the full context. Lookback Reranking recomputes the order within each block from that later position while keeping every head's budget unchanged, and it adds 1.0 to 3.0 points in all four settings.

Together, the three steps take the single-prefill score from 24.2–42.6 to 51.9–62.8, above KVzip in all four settings and without its second pass over the context.

→ Revised paper: Section 4.3, Table 2C (new).


AuthorsOpenReview ↗

Response to Reviewer KUoE (Weakness #2)

Weakness #2

Same allocation, different rankings

The first table gives (a) - KVzip with its own allocation, (b) - KVzip's ranking under RECAP-KV's allocation, (c) - our Base ranking under the same allocation, and (d) - the full RECAP-KV.

Model(a) KVzip, own alloc.(b) KVzip rank.(c) Base rank.(d) RECAP-KV
Qwen2.5-7B30% kept61.463.861.462.8
15% kept50.456.153.354.4
Time at 128K38.0 s38.0 s†20.1 s22.7 s
Llama-3.1-8B30% kept56.556.355.858.8
15% kept49.250.449.351.9
Time at 128K47.2 s47.2 s†25.6 s29.6 s
Qwen3.8-27B30% kept76.176.076.977.0
15% kept49.750.650.551.3
Time at 128K48.5 s48.5 s†25.3 s27.8 s

12-task mean score (0–100) over 873 contexts per model. Columns (b)–(d) (“RECAP-KV allocation”) keep as many KV pairs in each layer and head as RECAP-KV does and differ only in the ranking: KVzip's ranking, our Base ranking (the third row of the table in Weakness #1), and the full RECAP-KV (Base ranking + Lookback Reranking). Times are the prefill-and-scoring times of Table 1C: seconds to process one 128K-token context and compute the importance scores, before pruning and generation. †Same time as (a): KVzip's ranking needs its second pass under either allocation, and the allocation comes after the timed part. The Base-ranking times were measured in a separate run, in which RECAP-KV takes 22.8, 29.2 and 27.6 s. Bold: our columns (c) and (d) where they have the best score or the shortest time of the row.

How close is the Base score to KVzip?

With the allocation held fixed, the Base ranking, which needs no second pass, is 2.3 and 2.8 points below KVzip's ranking on Qwen2.5 and not distinguishable from it on the other two models: 0.5 and 1.1 points below on Llama-3.1, and 0.9 points above and 0.1 below on Qwen3.8-27B (all four intervals include zero). Lookback Reranking then narrows this distance or reverses it: the full RECAP-KV is 1.0 and 1.7 points below KVzip's ranking on Qwen2.5, 2.5 and 1.5 points above it on Llama-3.1, and equal within noise on Qwen3.8-27B.

KVzip under the same allocation

The second table splits RECAP-KV's margin over KVzip in Table 1 of the paper into the part due to the allocation and the part due to the ranking.

Difference (30% / 15%)Qwen2.5-7BLlama-3.1-8BQwen3.8-27B
Total: (d) − (a)+1.3 / +4.0+2.3 / +2.7+0.9 / +1.6
Allocation: (b) − (a)+2.3 / +5.7−0.2 / +1.2−0.1 / +0.8
Ranking: (d) − (b)−1.0 / −1.7+2.5 / +1.5+1.0 / +0.7

RECAP-KV's margin over KVzip, split into two steps; (a)–(d) are the columns of the first table. Allocation: KVzip's ranking, its own allocation replaced by RECAP-KV's. Ranking: same allocation, KVzip's ranking replaced by RECAP-KV's. Points. All differences in this answer are computed from unrounded scores.

The source of the margin differs by model: on Qwen2.5 it comes from the allocation, which also improves KVzip's own ranking by 2.3 and 5.7 points; on Llama-3.1 it comes from the ranking (the allocation part, −0.2 and +1.2, is not distinguishable from zero); and on Qwen3.8-27B both parts are small (−0.1 and +0.8 from the allocation, +1.0 and +0.7 from the ranking) and not distinguishable from zero.

The estimator's role

Both parts are components of RECAP-KV, and they have different jobs. The estimator is not meant to out-rank reconstruction. Its job is to reach reconstruction-level ranking without the second pass, and the table shows that it does: under identical budgets, RECAP-KV is between 1.7 points below and 2.5 points above KVzip's ranking, and it gets there in 37 to 43% less time (22.7 s against 38.0 s, 29.6 s against 47.2 s, and 27.8 s against 48.5 s at 128K).

→ Revised paper: Section 4.3, Table 3 (extended to three models).

AuthorsOpenReview ↗

Response to Reviewer KUoE (Question #1)

Question #1

Why the Base score sets the block budgets

RECAP-KV compresses the cache without using any query, so it cannot know where in the context a query's evidence lies. For this reason, the default takes its block budgets from a score that treats every position alike: the Base score views every block from the position right after it, at the same distance for every block. The Reranking score views every block from the end of the context, as reconstruction does. Within a block, where all tokens are at nearly the same distance, it gives an order closer to reconstruction than the Base score does (Table C2), and the default uses it for exactly that. Across blocks it ranks pairs near the end of the context higher, so when it also sets the budgets, the last 20% of the context keeps 17 to 53% more pairs on each of the nine tasks whose contexts are longer than 8K tokens, at the expense of earlier blocks (Llama-3.1, 15% retention; both configurations keep the same number of pairs in every head).

Effect on early evidence

On Llama-3.1, this lowers the accuracy of queries whose evidence lies at the beginning of the context when the budget is tight (see the table below, first row). In key–value retrieval and prefix–suffix copying, where every answer is a string that can be located in the context, the queries whose answer lies in the first 20% of the context are answered with 90.8% accuracy by the default and 84.9% by Reranking-only at 15% retention (5.8 points, computed from unrounded accuracies; 95% interval 2.4 to 9.1). At the other positions, and everywhere at 30% retention, the two are within about one point.

15% kept

Position of the answerQueriesFullKVRECAP-KVReranking-only
First 20% of the context29295.590.884.9
Middle 60%91791.484.284.3
Last 20%29190.483.584.5

30% kept

Position of the answerQueriesFullKVRECAP-KVReranking-only
First 20% of the context29295.595.294.5
Middle 60%91791.494.092.8
Last 20%29190.491.892.1

Accuracy (%) by the position of the answer in the context: Llama-3.1-8B-Instruct, key–value retrieval and prefix–suffix copying of Table 1 (200 contexts of about 18K Llama-3.1 tokens, 1,500 queries). RECAP-KV: the Base score sets the block budgets and the Reranking score orders the pairs within the blocks. Reranking-only: the Reranking score does both. Bold: the one cell where the two differ by more than about one point.

Reranking-only as a fast option

We keep the two-view configuration as the default because its block budgets come from a score that treats every position alike. On the 12-task mean the two configurations differ by 0.0 to 0.5 points at both retention levels, and all four 95% intervals include zero (the table of 12-task means). We therefore offer Reranking-only as a fast option, prompted by the reviewer's question: skipping the Base computation, which is the larger part of the scoring cost, cuts the construction time by 40 to 42%.

Configuration12-task mean score, Qwen2.5-7B, 30%12-task mean score, Qwen2.5-7B, 15%12-task mean score, Llama-3.1-8B, 30%12-task mean score, Llama-3.1-8B, 15%Cache construction time (s), Qwen2.5-7B, 32KCache construction time (s), Qwen2.5-7B, 64KCache construction time (s), Qwen2.5-7B, 128K
FullKV (ordinary prefill, no compression)65.765.761.261.21.033.019.97
RECAP-KV, default (two views)62.854.458.851.93.278.1823.81
RECAP-KV, fast option (Reranking-only)62.854.758.351.41.964.8713.76

RECAP-KV and its fast option. Scores: 12-task means (0–100) over 873 contexts per model at 30% and 15% retention. Time (Qwen2.5-7B): from token IDs to the compressed cache at 30% retention on one B300 GPU, median of 5 runs after 2 warm-ups; 128K = 130,816 tokens; the FullKV row is the prefill alone. Bold: the shorter construction time of the two RECAP-KV configurations.

→ Revised paper: Appendix E, Tables E1 and E2 (new): Reranking-only as a fast option; the default is unchanged.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.