acceptodds
Under review as a conference paper at ICLR 2027

KVScout: Efficient Long-Context Decoding with Sentence-Level KV Retrieval

Abstract

Long-context models show strong capabilities in processing long inputs but incur substantial memory and computational costs due to the growing KV cache. KV retrieval methods reduce these costs by keeping the full context KV cache in CPU memory and retrieving only a small subset to the GPU for attention. However, frequent KV selection and transfer can offset sparse-attention savings and limit decoding speedup, motivating fewer retrievals during decoding. Yet, existing selectors are designed for next-token retrieval, and reusing their selections across more decoding steps degrades generation quality. To address these limitations, we propose **KVScout**, an efficient KV retrieval method that learns to select the KV entries needed for generating the upcoming sentence. With a lightweight retriever (8.9M parameters), KVScout builds a contextualized sentence index during prefill and queries it with the generation state before generating each sentence, enabling each retrieved KV window to be reused throughout that sentence. Extensive experiments on three long-context benchmarks demonstrate that KVScout maintains generation quality with a low refresh frequency, achieving 5.3× and 1.7–3.7× higher throughput than full-cache decoding and existing KV retrieval methods at 256K context, respectively.

Primary area:
foundation or frontier models, including LLMs
Keywords:
KV Cache Retrieval, Efficient Long-Context Decoding

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_M56yOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

KV retrieval saves attention, but frequent retrieval creates new overhead, so this paper proposed a sentence-level retrieval method, which avoids repeated retreival within a generated sentence.

Critical strengths
  • I like the clear problem formulation.
  • For the sentence representation, this paper uses special trainable embeddings, which could be more accurate than simple pooling on the keys.
  • The hierarchical token selection method is smart: firstly retrieve important sentences, and then pick important tokens inside the retrieved sentences.
Critical weaknesses
  • No ablation of fixed-size chunk retrival.
  • The overall idea is built on SentenceKV's sentence-level offloading and onloading, but the paper did not mention what is the difference between SentenceKV and KVScout.
  • The paper focuses on decoding, not prefilling. But benchmarks like LongBench and RULER are long-input/short-output benchmarks. The reason I give this paper weak rejection is mainly because of this. I hope to see the results of reasoning benchmarks like MATH500 and GPQA-D.
  • With reasoning benchmark results and the answer to the question below, I'll raise my score.
Questions for authors

I feel like there is a potential mismatch between the training and inference of the [RET] probe, and hope the author can give some explanations for this.

During training, the paper states that the [RET] probe attends to the full input context together with the generation prefix up to the current sentence boundary. However, during inference, the full context KV cache has been offloaded, and the [RET] probe only attends to the current generation prefix and the GPU-resident window Wt=S∪DtW_t = S \cup D_t. Therefore, the retrieval query is constructed under different information conditions during training and inference. This raises two concerns. First, during training, the [RET] probe may directly attend to the relevant evidence that it is subsequently trained to retrieve, which could make the contrastive retrieval task easier than the actual predictive retrieval problem encountered at inference. Second, this train-test distribution mismatch may affect the reliability of the reported retriever accuracy if the validation queries are also constructed with access to the full context. It would be helpful for the authors to clarify this discrepancy and evaluate a training setting in which the [RET] probe is conditioned only on a simulated sparse window matching the inference-time WtW_t. An ablation comparing full-context training against inference-matched sparse-window training would help determine how much this mismatch affects retrieval quality and downstream generation.

Reviewer_aPm7OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents a long-context decoding method called KVScout. It retrieves KV entries once per generated sentence rather than once per token, and uses learned sentence representations to predict relevant context, followed by token-level refinement and reuse of the resulting KV window. Experiments on three long-context benchmarks demonstrate quality and substantial throughput improvements, including 5.3x speedup over full-cache decoding at 256k context.

Critical strengths
  1. Clear motivation: the paper identifies KV overhead and CPU-GPU transfer as important bottlenecks.

  2. Lightweight design: the base LLM remains frozen, and only an 8.9M parameter retriever is trained.

  3. The reported quality-efficiency tradeoff is strong. It achieves the best average LongBench score among the evaluated compressed methods across both models and all tested budget.

Critical weaknesses
  1. SentenceKV is absent form the throughput and latency comparisons, which is the most closely related method. This weakens the claim that KVScout is the most efficient sentence level retrieval approach.

  2. The reported decoding speedups do not clearly account for sentence segmentation, [SUM] index construction, additional prefill computation, index storage, or transferring oversized candidate sentences before token refinement.

  3. Evaluations focus on long-context QA and retrieval tasks, but do not include other domains, languages, code, and irregular sentence structures unclear.

  4. The comparisons mix retrieval granularity with the selection method, so it remains unclear whether the gains come form sentence segmentation, predictive retrieval or overall design. A controlled ablation is needed.

Questions for authors
  1. Can you provide a controlled comparison of sentence, block, and page level retrieval using the same predictor, budget and refresh frequency.

  2. Why is SentenceKV excluded from the throughput and latency comparisons?

  3. How accurate if the automatically generated supervision, and does a retriever trained on HotpotQA transfer to code, tables and non-QA workloads?

  4. How were retrieval layers 24 and 21 selected, and how sensitive is performance to this choice?

Reviewer_uYwoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes KVScout to accelerate long-context decoding with CPU-offloaded KV caches. KVScout learns to retrieve the context needed for the upcoming sentence and reuses the selected KV entries throughout its generation. The method combines sentence-level indexing, a lightweight trainable retriever, a persistent KV set, and token-level refinement. Experiments on 3 long-context benchmarks show that KVScout achieving 5.3× and 1.7–3.7× higher throughput than full-cache decoding and existing KV retrieval methods at 256K context, respectively

Critical strengths

1.The method is well designed. KVScout learns to predict the context needed for the upcoming sentence and reuses the retrieved KV entries throughout its generation. This design directly addresses the overhead of frequent KV selection and transfer while maintaining access to relevant context. 2.Experiments demonstrate the effectiveness of KVScout. On Qwen3-8B, KVScout achieves a 5.3× decoding throughput speedup over full-cache decoding and a 1.74× speedup over Quest at 256K context.

Critical weaknesses
  1. The end-to-end efficiency evaluation could be more comprehensive. Figure 4 reports decoding throughput improvements but does not account for the overhead of sentence indexing and KV offloading during prefill. Reporting end-to-end latency would better demonstrate the practical efficiency gains.
  2. The throughput comparison could be strengthened by including SentenceKV. Although SentenceKV is included in the generation quality evaluation (Table 1), it is absent from the throughput comparisons (Figures 4、5). Including this baseline or explaining its omission would strengthen the efficiency evaluation.
Questions for authors

1.Could the authors report end-to-end latency, including the overhead of sentence indexing and KV offloading during prefill? 2.Could the authors include SentenceKV in the throughput comparison or explain why it was omitted?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.