- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes KV-Fold, a method that performs per-chunk transformation on the KV-cache. It identifies the standard chunked prefill as the identity case of KV-Fold, in which the KV cache state is carried unchanged across chunks. The paper proves the equivalence and verifies the equivalence empirically. It also studies non-identity transformations including quantization, selective preservation, and query-blind selection, in a single producer-consumer stream, as well as transfers of KV-cache states between separate computations. In general, the evaluation results show that through KV-Fold, identity transformations yield next-token loss matching full attention at lower memory costs and that non-identity transformations allow for various flexible design choices.
Critical strengths
S1. The paper views uniformly a set of common operations on KV-caches in long-context inference tasks in pretrained transformers as transformations and transfers between KV-cache states, proposes per-chunk transformations to reduce the peak memory requirement and allow for flexible design choices.
S2. The paper studies the equivalence of identity transformations with the proposed approach as well as evaluates various design choices of the non-identity transformations and transfers.
Critical weaknesses
W1. The paper claims that KV-caches are usually treated as an internal artifact of attention whereas the authors view the KV cache as an explicit computational state. I don't really see the fundamental difference between the two. Internal artifacts can be manipulated through transformations and can also be made external (exposed) between source and target models, even if we don't call them messages/states, as the paper does. This kind of views in the paper sometimes feel more confusing than clarifying, as we need to track a number of terms - KV states, computational states, recurrent interface, manipulatable state interface, sparse messages between separate inference processes, and so on.
W2. While this is an interesting study on an alternative method of manipulating KV-caches for long-context inference, the key contribution is the per-chunk transformation. It can save peak memory consumption and support accesses to contents at longer distances, but it also bears some inherent limitations, e.g., slower prefill, intolerance of systematic information removal, and inability to identify content for query-blind selection. Also, the paper shows a variety of design choices possible with KV-fold; however, these choices come with different tradeoffs and it is unclear how these choices can be selected in practice without a lot of tuning effort.
Questions for authors
Q1.This paper proposes an alternative view of KV-cache manipulation and studies several KV-cache transformation tasks with KV-Fold. It is nice that the paper gives comprehensive evaluations and analysis on the pros and cons, advantages and limitations of the proposed approach in comparison with existing work. However, in the end, I am still unclear about what major technical problem this paper solves and what practical impact it makes. After all, KV-caches are heavily used in real systems for end-to-end performance. I would like to hear from the authors on the systems aspect (which KV-caches are designed for), other than the threefold contributions listed in the paper.
5 comments
Major Technical Contribution
Response From a systems perspective, KV-Fold addresses two main challenges: making long-context inference feasible under limited GPU memory and enabling KV states to be reused, manipulated, and transferred between separate computations. Our incremental-retention experiment provides measured construction-memory results. With Llama-3.1-8B-Instruct in FP16, processing inputs of 512 to 13,380 tokens in 64-token chunks while retaining the most recent 512 KV positions kept the persistent cache at 64 MiB and peak GPU allocation, including model weights, at 16.27–16.32 GB over the tested lengths. This measures construction memory without establishing preserved task accuracy.
Our additional experiments further demonstrate the value of state reuse. For each paper, the model built a complete document cache without access to the questions. We reused these caches across twelve questions from five papers, with every cache-based continuation matching the continuation produced by the full-text baseline. This agreement also held for 24 additional wording variants. We also show that question-specific checkpoints can preserve ongoing reasoning while reducing receiving time compared with reconstructing the same state through sequential text replay.
The reviewer's question also motivated us to explore a related theoretical question: under what conditions can a fixed, sparse KV state remain useful for previously unseen queries? We derived a bound showing that sparse attention can approximate full attention when the retained positions capture sufficient attention mass and new queries remain close to the original query in attention-query space. The bound also explains why this approximation may deteriorate as queries become more dissimilar.
Proof sketch. Bounded key norms limit the effect of changing the query on the attention weights. The output error is then controlled by the weight assigned to discarded positions and the value norms. Section 3 of the mathematical supplement gives the full theorem and proof.
To examine this result empirically, we studied a frozen attention head in Qwen3-8B. Retaining just 14.43% of the source KV positions preserved approximately 78–79% of the source-normalized attention mass across three query vectors, with the observed attention-output errors remaining below the theoretical bounds. While this is not a guarantee of downstream accuracy, it offers initial evidence for the mechanism described by our analysis.
Taken together, we believe these results clarify the technical contribution and practical motivation behind KV-Fold. Beyond making memory-constrained long-context inference possible, KV-Fold provides a common framework for understanding how computational states can be preserved, manipulated, and shared across inference processes. Although building a fully optimized serving system is beyond the scope of this paper, we see these capabilities as a useful foundation for future long-context and multi-process inference systems.
Restoring a checkpoint versus rebuilding from text. This comparison uses the same thirty continuations from the fourteen papers that completed the expansion, with the model already loaded on an H100. Exact text replay is the baseline and rebuilds the state using the original source-prefill and token-by-token transcript schedule. Fast text replay instead uses 512-token chunks. Our method restores the saved KV checkpoint.
Method Exact continuation matches Receiving time (s) Estimated cost (USD)
---------------------------- -------------------------- ------------------ --------------------
Exact text replay (baseline) 30/30 383.33 0.37162
Fast chunked text replay 13/30 237.66 0.23040
KV checkpoint restoration 30/30 227.44 0.22050
Exact continuation matches require identical generated tokens, final text, EOS and sampling state to the original run. These counts measure continuation fidelity, not answer correctness. Thus, 13/30 does not mean that seventeen answers were wrong, and 30/30 does not imply perfect answer accuracy. A different continuation can still produce a correct answer. Numerical logit fidelity is evaluated separately.
Answer quality is scored separately. We first averaged question scores within each paper, then averaged across papers. The resulting answer and evidence F1 scores were 0.446 and 0.307 for exact replay and checkpoints, compared with 0.419 and 0.292 for fast replay.
Restoring the checkpoint used 40.67% less receiving time than the exact text replay baseline, reducing the estimated cost of the receiving computation by USD 0.15112 across these thirty continuations. Times are summed across questions and include state reconstruction or restoration and subsequent decoding with the same logging and output-recording procedures in all three methods. The geometric mean of the per-paper time ratios gives a 47.85% reduction (95% paired-paper bootstrap interval of 40.33–55.76%), conditional on the fourteen completed paper pairs. The timing difference between checkpoints and fast text replay is inconclusive, with the paired-paper 95% interval including no advantage.
Estimated costs use the recorded H100 rate of USD 3.49 per GPU-hour and unrounded times. They cover only the receiving computation, not the total billed cost. Model loading, work done by the sending process, writing, CPU-side storage and handling, transport, storage and idle time are accounted for separately. The complete two-host finish-and-relay comparison is reported separately in the appendix because it measures end-to-end answer delivery on a single question.
All rows start from a KV cache built by processing the same complete 32,768-token input. The table compares the KV memory retained afterward, across twenty trials using six reused documents.
Cache representation Retained KV memory (MiB) KV memory saved vs. full cache
----------------------------------- ------------------------ ------------------------------
Full cache (baseline) 4,096.25 0%
KV-Fold selection 4.50 99.89%
KV-Fold selection and re-addressing 20.88 99.49%
After processing the complete document, we run the target question against its cache and rank document positions by mean attention across question tokens, heads and layers. We retain the highest-scoring positions, always including the first four positions within the budget, and keep their key and value vectors across all layers and heads. The model then answers using the reduced cache. The two variants retain 36 and 167 positions, respectively. Re-addressing additionally moves the retained positions to consecutive slots and adjusts their RoPE rotations. The 36-position variant reduces retained KV memory from approximately 4 GiB to 4.50 MiB, using 99.89% less memory while preserving all twenty correct target answers. This matters when GPU memory available for the cache is limited after loading model weights and reserving working memory. For example, with a 1 GiB budget for retained KV state, the full cache exceeds the budget while both KV-Fold representations fit. This capacity comparison is calculated from KV tensor and position-index sizes.
We thank the reviewer for these helpful suggestions. To address the concerns raised, we have substantially expanded both the scale and scope of the empirical evaluation. For example, the checkpoint study now recovers 42 continuations across 19 completed papers exactly, compared with twelve continuations across five papers in the initial study. We have also added paraphrase evaluations, text and retrieval baselines, cross-GPU continuation tests, cost comparisons and measurements of memory use during cache construction.
Supporting results, analysis scripts and reproduction code are available in our anonymous supplementary repository.
We thank the reviewer for their constructive feedback. We address the main concerns below.
Internal vs Computational space.
Response We agree that KV caches can already be manipulated, regardless of whether they are viewed as internal artifacts or computational states. However, we show that the distinction is more than just terminology. By treating the KV cache as an explicit computational state, we can reason about how this state evolves, observe how different transformations affect it, and potentially apply tools from control theory to analyze and manipulate it.
This perspective is particularly useful because it separates the model's underlying computation from the operations performed on its state. Rather than treating quantization, selective retention, and state transfer as unrelated techniques, we can study them within a common framework and better understand their effects on subsequent computation. Further, our results show practical outcomes in both reuse across multiple questions of a complete document cache built without access to those questions and exact resumption of question-specific reasoning checkpoints. These are separate experiments with different information boundaries.
We believe this framing also opens up interesting directions for future work, particularly in designing state transformations using ideas from control theory. We will clarify this motivation and simplify the terminology in the revised paper.
Tradeoffs
Response One of the goals of KV-Fold is to provide a common framework for understanding how different state transformations affect information preservation, memory usage, and computational cost.
The identity transformation serves as a reference for exact preservation, while selective retention, quantization, and re-addressing introduce different tradeoffs. Our additional experiments further clarify these distinctions. Reusing a complete document cache built without access to the questions preserves the matched text continuation across multiple questions, while question-specific checkpoints preserve ongoing reasoning and reduce receiving time compared with exact sequential text reconstruction.
The appropriate transformation depends on the application's requirements, and we do not claim that one approach is optimal in every setting. Rather, KV-Fold makes these tradeoffs explicit and allows them to be evaluated systematically. We have also expanded the comparisons and cost analysis in the paper and appendix to make this clearer.