acceptodds
Under review as a conference paper at ICLR 2027

KV-Fold: Beyond Chunked Prefill with Manipulable KV State

Abstract

Long-context inference in pretrained transformers relies on a growing key–value (KV) cache that is usually treated as an internal artifact of attention. We instead view the KV cache as an explicit computational state that can be carried, modified, and transferred between computations. Standard chunked prefill is the identity case of this view: carrying the KV state unchanged across chunks is exactly equivalent, in exact arithmetic, to a monolithic causal forward pass. We prove this equivalence and verify it across several pretrained transformer families, with next-token loss matching full attention to within nats. This exact identity provides a controlled reference point for studying what happens when the carried state is changed. Without modifying model parameters, we quantize, selectively retain, re-address, and transfer KV state. The unmodified state preserves exact factual retrieval across contexts up to K tokens and chunk transitions while reducing peak working memory enough to process a context for which the corresponding monolithic forward does not fit on the same hardware. Under lossy transformations, per-step quantization reveals a graded memory–fidelity trade-off, whereas position removal is substantially less forgiving. Most strikingly, KV state can serve as a sparse message between otherwise separate inference processes. At K tokens, a consumer that never reads the original document recovers a designated fact in all trials from a shared subset of only of cache positions; matched-bandwidth random and recency controls recover none. A per-head question-aware selector succeeds even at retained positions, and RoPE-correct re-addressing allows selected state to be moved into a new positional coordinate frame without loss of retrieval. These results suggest that the KV cache is not merely stored context, but a manipulable state interface through which pretrained transformers can preserve, compress, address, and communicate information without architectural changes or fine-tuning.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Persistent KV State, Stateful Inference, Long-Context Inference

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Qwc5OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes KV-Fold, a method that performs per-chunk transformation on the KV-cache. It identifies the standard chunked prefill as the identity case of KV-Fold, in which the KV cache state is carried unchanged across chunks. The paper proves the equivalence and verifies the equivalence empirically. It also studies non-identity transformations including quantization, selective preservation, and query-blind selection, in a single producer-consumer stream, as well as transfers of KV-cache states between separate computations. In general, the evaluation results show that through KV-Fold, identity transformations yield next-token loss matching full attention at lower memory costs and that non-identity transformations allow for various flexible design choices.

Critical strengths

S1. The paper views uniformly a set of common operations on KV-caches in long-context inference tasks in pretrained transformers as transformations and transfers between KV-cache states, proposes per-chunk transformations to reduce the peak memory requirement and allow for flexible design choices.

S2. The paper studies the equivalence of identity transformations with the proposed approach as well as evaluates various design choices of the non-identity transformations and transfers.

Critical weaknesses

W1. The paper claims that KV-caches are usually treated as an internal artifact of attention whereas the authors view the KV cache as an explicit computational state. I don't really see the fundamental difference between the two. Internal artifacts can be manipulated through transformations and can also be made external (exposed) between source and target models, even if we don't call them messages/states, as the paper does. This kind of views in the paper sometimes feel more confusing than clarifying, as we need to track a number of terms - KV states, computational states, recurrent interface, manipulatable state interface, sparse messages between separate inference processes, and so on.

W2. While this is an interesting study on an alternative method of manipulating KV-caches for long-context inference, the key contribution is the per-chunk transformation. It can save peak memory consumption and support accesses to contents at longer distances, but it also bears some inherent limitations, e.g., slower prefill, intolerance of systematic information removal, and inability to identify content for query-blind selection. Also, the paper shows a variety of design choices possible with KV-fold; however, these choices come with different tradeoffs and it is unclear how these choices can be selected in practice without a lot of tuning effort.

Questions for authors

Q1.This paper proposes an alternative view of KV-cache manipulation and studies several KV-cache transformation tasks with KV-Fold. It is nice that the paper gives comprehensive evaluations and analysis on the pros and cons, advantages and limitations of the proposed approach in comparison with existing work. However, in the end, I am still unclear about what major technical problem this paper solves and what practical impact it makes. After all, KV-caches are heavily used in real systems for end-to-end performance. I would like to hear from the authors on the systems aspect (which KV-caches are designed for), other than the threefold contributions listed in the paper.

5 comments

We thank the reviewer for their constructive feedback. We address the main concerns below.

Internal vs Computational space.

Response We agree that KV caches can already be manipulated, regardless of whether they are viewed as internal artifacts or computational states. However, we show that the distinction is more than just terminology. By treating the KV cache as an explicit computational state, we can reason about how this state evolves, observe how different transformations affect it, and potentially apply tools from control theory to analyze and manipulate it.

This perspective is particularly useful because it separates the model's underlying computation from the operations performed on its state. Rather than treating quantization, selective retention, and state transfer as unrelated techniques, we can study them within a common framework and better understand their effects on subsequent computation. Further, our results show practical outcomes in both reuse across multiple questions of a complete document cache built without access to those questions and exact resumption of question-specific reasoning checkpoints. These are separate experiments with different information boundaries.

We believe this framing also opens up interesting directions for future work, particularly in designing state transformations using ideas from control theory. We will clarify this motivation and simplify the terminology in the revised paper.

Tradeoffs

Response One of the goals of KV-Fold is to provide a common framework for understanding how different state transformations affect information preservation, memory usage, and computational cost.

The identity transformation serves as a reference for exact preservation, while selective retention, quantization, and re-addressing introduce different tradeoffs. Our additional experiments further clarify these distinctions. Reusing a complete document cache built without access to the questions preserves the matched text continuation across multiple questions, while question-specific checkpoints preserve ongoing reasoning and reduce receiving time compared with exact sequential text reconstruction.

The appropriate transformation depends on the application's requirements, and we do not claim that one approach is optimal in every setting. Rather, KV-Fold makes these tradeoffs explicit and allows them to be evaluated systematically. We have also expanded the comparisons and cost analysis in the paper and appendix to make this clearer.

Major Technical Contribution

Response From a systems perspective, KV-Fold addresses two main challenges: making long-context inference feasible under limited GPU memory and enabling KV states to be reused, manipulated, and transferred between separate computations. Our incremental-retention experiment provides measured construction-memory results. With Llama-3.1-8B-Instruct in FP16, processing inputs of 512 to 13,380 tokens in 64-token chunks while retaining the most recent 512 KV positions kept the persistent cache at 64 MiB and peak GPU allocation, including model weights, at 16.27–16.32 GB over the tested lengths. This measures construction memory without establishing preserved task accuracy.

Our additional experiments further demonstrate the value of state reuse. For each paper, the model built a complete document cache without access to the questions. We reused these caches across twelve questions from five papers, with every cache-based continuation matching the continuation produced by the full-text baseline. This agreement also held for 24 additional wording variants. We also show that question-specific checkpoints can preserve ongoing reasoning while reducing receiving time compared with reconstructing the same state through sequential text replay.

The reviewer's question also motivated us to explore a related theoretical question: under what conditions can a fixed, sparse KV state remain useful for previously unseen queries? We derived a bound showing that sparse attention can approximate full attention when the retained positions capture sufficient attention mass and new queries remain close to the original query in attention-query space. The bound also explains why this approximation may deteriorate as queries become more dissimilar.

Proof sketch. Bounded key norms limit the effect of changing the query on the attention weights. The output error is then controlled by the weight assigned to discarded positions and the value norms. Section 3 of the mathematical supplement gives the full theorem and proof.

To examine this result empirically, we studied a frozen attention head in Qwen3-8B. Retaining just 14.43% of the source KV positions preserved approximately 78–79% of the source-normalized attention mass across three query vectors, with the observed attention-output errors remaining below the theoretical bounds. While this is not a guarantee of downstream accuracy, it offers initial evidence for the mechanism described by our analysis.

Taken together, we believe these results clarify the technical contribution and practical motivation behind KV-Fold. Beyond making memory-constrained long-context inference possible, KV-Fold provides a common framework for understanding how computational states can be preserved, manipulated, and shared across inference processes. Although building a fully optimized serving system is beyond the scope of this paper, we see these capabilities as a useful foundation for future long-context and multi-process inference systems.

Restoring a checkpoint versus rebuilding from text. This comparison uses the same thirty continuations from the fourteen papers that completed the expansion, with the model already loaded on an H100. Exact text replay is the baseline and rebuilds the state using the original source-prefill and token-by-token transcript schedule. Fast text replay instead uses 512-token chunks. Our method restores the saved KV checkpoint.

Method                        Exact continuation matches  Receiving time (s)  Estimated cost (USD)
----------------------------  --------------------------  ------------------  --------------------
Exact text replay (baseline)                       30/30              383.33               0.37162
Fast chunked text replay                           13/30              237.66               0.23040
KV checkpoint restoration                          30/30              227.44               0.22050

Exact continuation matches require identical generated tokens, final text, EOS and sampling state to the original run. These counts measure continuation fidelity, not answer correctness. Thus, 13/30 does not mean that seventeen answers were wrong, and 30/30 does not imply perfect answer accuracy. A different continuation can still produce a correct answer. Numerical logit fidelity is evaluated separately.

Answer quality is scored separately. We first averaged question scores within each paper, then averaged across papers. The resulting answer and evidence F1 scores were 0.446 and 0.307 for exact replay and checkpoints, compared with 0.419 and 0.292 for fast replay.

Restoring the checkpoint used 40.67% less receiving time than the exact text replay baseline, reducing the estimated cost of the receiving computation by USD 0.15112 across these thirty continuations. Times are summed across questions and include state reconstruction or restoration and subsequent decoding with the same logging and output-recording procedures in all three methods. The geometric mean of the per-paper time ratios gives a 47.85% reduction (95% paired-paper bootstrap interval of 40.33–55.76%), conditional on the fourteen completed paper pairs. The timing difference between checkpoints and fast text replay is inconclusive, with the paired-paper 95% interval including no advantage.

Estimated costs use the recorded H100 rate of USD 3.49 per GPU-hour and unrounded times. They cover only the receiving computation, not the total billed cost. Model loading, work done by the sending process, writing, CPU-side storage and handling, transport, storage and idle time are accounted for separately. The complete two-host finish-and-relay comparison is reported separately in the appendix because it measures end-to-end answer delivery on a single question.

All rows start from a KV cache built by processing the same complete 32,768-token input. The table compares the KV memory retained afterward, across twenty trials using six reused documents.

Cache representation                 Retained KV memory (MiB)  KV memory saved vs. full cache
-----------------------------------  ------------------------  ------------------------------
Full cache (baseline)                                4,096.25                              0%
KV-Fold selection                                        4.50                          99.89%
KV-Fold selection and re-addressing                     20.88                          99.49%

After processing the complete document, we run the target question against its cache and rank document positions by mean attention across question tokens, heads and layers. We retain the highest-scoring positions, always including the first four positions within the budget, and keep their key and value vectors across all layers and heads. The model then answers using the reduced cache. The two variants retain 36 and 167 positions, respectively. Re-addressing additionally moves the retained positions to consecutive slots and adjusts their RoPE rotations. The 36-position variant reduces retained KV memory from approximately 4 GiB to 4.50 MiB, using 99.89% less memory while preserving all twenty correct target answers. This matters when GPU memory available for the cache is limited after loading model weights and reserving working memory. For example, with a 1 GiB budget for retained KV state, the full cache exceeds the budget while both KV-Fold representations fit. This capacity comparison is calculated from KV tensor and position-index sizes.

We thank the reviewer for these helpful suggestions. To address the concerns raised, we have substantially expanded both the scale and scope of the empirical evaluation. For example, the checkpoint study now recovers 42 continuations across 19 completed papers exactly, compared with twelve continuations across five papers in the initial study. We have also added paraphrase evaluations, text and retrieval baselines, cross-GPU continuation tests, cost comparisons and measurements of memory use during cache construction.

Supporting results, analysis scripts and reproduction code are available in our anonymous supplementary repository.

Reviewer_JFeBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes KV fold, which studies how KV caches can be modified and reused when processing long contexts in chunks. Experiments show that preserving the full KV cache reduces peak memory without affecting computation or long-range retrieval. Quantization gradually degrades performance, while a small, query-selected subset of KV positions can retain the target information even after positional re-indexing.

Critical strengths
  1. The paper is clearly written, and the KV-Fold setup makes the experiments easy to follow.

  2. The equivalence result is supported by both a proof and experiments on several models.

Critical weaknesses
  1. The novelty is limited, as both KV quantization and attention-based KV cache selection have been studied before.

  2. Most experiments focus on retrieving a five-digit number and use relatively few trials. The Qwen results are also unstable, so it is unclear whether the findings generalize to more complex tasks.

  3. The paper includes a SnapKV-style baseline but lacks broader end-to-end comparisons with other KV-cache compression methods.

Questions for authors
  1. What is the main technical novelty of KV-Fold beyond combining chunked prefill with existing KV-cache compression methods?
4 comments

We thank the reviewer for their constructive feedback. We address the main concerns below.

Novelty and Technical Contribution.

Response We agree that KV quantization and attention-based selection are established techniques. Our technical contribution is a common state-transition formulation with an exact identity reference and conditional analysis, used to separate preservation of a computation from the effects of selection, re-addressing and transfer. The checkpoint experiments provide a controlled test of that distinction within the framework.

Inspired by the foldl (left-fold) abstraction in functional programming, KV-Fold separates the frozen transformer from transformations applied to its evolving state. At each step, a state transformation is applied before the frozen transformer processes the next input. We call this perspective KV-Space Programming, drawing a conceptual connection to recent Weight-Space Programming. Rather than modifying model parameters, we investigate how manipulating inference-time state can influence subsequent computations.

Importantly, this perspective also admits theoretical analysis. We can show that, under bounded key and value norms, the error introduced by sparse KV selection is controlled by the attention mass assigned to discarded positions. Furthermore, this guarantee extends to unseen queries that are sufficiently close in attention-query space to the original query. This conditional guarantee concerns attention outputs rather than answer accuracy for a complete transformer.

Our experiments illustrate the potential of this interface. The separately recovered sparse reproduction records 20/20 correct 36-position trials over six reused 32K-token documents. More broadly, the expanded reasoning-checkpoint experiments recover 42 continuations across 19 papers exactly, with a 47.85% reduction in receiving time versus exact text replay across the fourteen completed expansion pairs (95% paired-paper bootstrap interval of 40.33–55.76%). These demonstrate complementary uses of the state interface through selective factual access and exact computation recovery.

Generalization and Comparisons.

Response We thank the reviewer for this thoughtful question, as it sparked theoretical curiosity in the authors.

Our analysis reveals an encouraging conditional result for a frozen attention head. A fixed sparse KV state can approximate the full attention output for multiple previously unseen queries, provided they are sufficiently close to the original query in attention-query space and the retained positions capture enough of the relevant attention mass. This is not by itself a guarantee of downstream answer accuracy.

Empirically, for each paper, the model built a cache from the full document without access to the questions. We reused these caches for twelve original questions across five papers. All twelve comparisons with the full-text baseline agreed in tokens, text, EOS and the run's random-number generator state. Twenty-four paraphrases fixed before testing also yielded matching pairs. All 96 paraphrase and method combinations reached EOS. These tests reuse the complete cache over the same five papers. The paraphrase tests were designed after the original questions had been evaluated. The sparse attention theorem addresses a separate conditional setting.

Importantly, the result also characterizes when this guarantee may break down. As downstream queries become more dissimilar, attention may shift toward discarded positions, weakening the approximation guarantee.

For example, separate receiving processes answered the three fixed CrossWOZ questions using the same cache. This cache was built from the full document without access to the questions, and the receiving processes never read the original document. This illustrates reusable source evidence. It does not verify the sparse theorem's query-distance premise.

We also examined this mechanism in one frozen Qwen3-8B attention head using source-only KV state. A fixed, uniformly selected subset of 2,048 out of 14,195 positions (14.43%) retained 78–79% of the source-normalized attention mass across three query vectors. These vectors were reconstructed from hidden states recorded during model inference, using the model's own query projection, normalization and RoPE. L2 errors of 0.157–0.166 in this source-only attention calculation lay below the corresponding bounds of 4.50–4.67. The bounds are conservative. This is a single-head mechanism check, with downstream utility requiring separate validation.

Proof sketch. A small change in the query produces a bounded change in each attention score because the key norms are bounded. This limits how much attention can shift to discarded positions. The full attention output is a weighted average of the outputs from retained and discarded positions. Their difference is bounded by twice the largest value norm, so the error is controlled by the attention mass discarded. Combining these observations gives the query-distance bound. The complete theorem, assumptions and proof are in Section 3 of the mathematical supplement.

To address the reviewer's concern about generalization beyond single-needle retrieval, we added document question-answering experiments to our existing PG-19 language-modeling evaluation. We tested twelve questions from five QASPER papers under four methods, yielding 48 question and method combinations. Transfer of the complete KV cache matched the full-text baseline on all twelve continuations.

We also tested 24 paraphrases under the same four methods, yielding 96 combinations. Every cache-based continuation matched the full-text baseline. The paraphrases used the same five papers and were designed after the original questions had been evaluated, then fixed before testing.

A separate checkpoint study added fourteen completed paper pairs and the preregistered paper-level uncertainty analysis described above. Its confidence interval concerns the reduction in checkpoint receiving time relative to exact text replay. It does not apply to the earlier exploratory ten-trial selective-quantization results.

Regarding comparisons, we already evaluate against StreamingLLM, SnapKV-style selection, and matched-budget random and recency baselines. The expanded evaluation adds BM25 (16 KiB), no-context, the full-text baseline, text replay using the original processing schedule and fast chunked text replay. The reasoning-checkpoint study additionally compares exact continuation with measured receiving time and its estimated cost at the recorded rental rate for the initial five-paper study and the new fourteen-paper completed cohort, as reported below. These are state transfer and reconstruction comparisons. Matched-memory comparisons with optimized compressed-cache methods remain a separate empirical gap. The appendix also reports our additional native Qwen3 SnapKV adaptation and its qualification limits, separately from the original SnapKV-style baseline.

For the fourteen completed expansion papers and thirty continuations, the table compares checkpoint restoration with two text-replay methods on an H100, with the model already loaded. Exact text replay uses the original processing schedule and is the baseline.

Method                        Exact continuation matches  Receiving time (s)  Estimated cost (USD)
----------------------------  --------------------------  ------------------  --------------------
Exact text replay (baseline)                       30/30              383.33               0.37162
Fast chunked text replay                           13/30              237.66               0.23040
KV checkpoint restoration                          30/30              227.44               0.22050

Times sum state restoration or reconstruction and generation. Continuation matches measure agreement with the original run, not answer correctness. Dollar estimates cover only this receiving computation at USD 3.49 per GPU-hour, excluding model loading, writing, transport and other costs. The timing difference between checkpoints and fast replay is inconclusive.

All rows start from a KV cache built by processing the same complete 32,768-token input. The table compares the KV memory retained afterward, across twenty trials using six reused documents.

Cache representation                 Retained KV memory (MiB)  KV memory saved vs. full cache
-----------------------------------  ------------------------  ------------------------------
Full cache (baseline)                                4,096.25                              0%
KV-Fold selection                                        4.50                          99.89%
KV-Fold selection and re-addressing                     20.88                          99.49%

After processing the complete document, we run the target question against its cache and rank document positions by mean attention across question tokens, heads and layers. We retain the highest-scoring positions, always including the first four positions within the budget, and keep their key and value vectors across all layers and heads. The model then answers using the reduced cache. The two variants retain 36 and 167 positions, respectively. Re-addressing additionally moves the retained positions to consecutive slots and adjusts their RoPE rotations. The 36-position variant reduces retained KV memory from approximately 4 GiB to 4.50 MiB, using 99.89% less memory while preserving all twenty correct target answers. This matters when GPU memory available for the cache is limited after loading model weights and reserving working memory. For example, with a 1 GiB budget for retained KV state, the full cache exceeds the budget while both KV-Fold representations fit. This capacity comparison is calculated from KV tensor and position-index sizes.

We thank the reviewer for these helpful suggestions. To address the concerns raised, we have substantially expanded both the scale and scope of the empirical evaluation. For example, the checkpoint study now recovers 42 continuations across 19 completed papers exactly, compared with twelve continuations across five papers in the initial study. We have also added paraphrase evaluations, text and retrieval baselines, cross-GPU continuation tests, cost comparisons and measurements of memory use during cache construction.

Supporting results, analysis scripts and reproduction code are available in our anonymous supplementary repository.

As the discussion period draws to a close, we would like to thank the reviewer again for their time and thoughtful feedback. We hope our responses and additional results have helped address your concerns. If anything remains unclear, or if there are further experiments you would find useful, we would be happy to address them before the discussion ends. We appreciate your feedback and look forward to hearing any final thoughts.

Reviewer_dvSCOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper presents KV-Fold, a framework treating the KV cache as a recurrent computational state that can be manipulated between successive chunks. The authors formulate standard chunked prefill as the identity (g=idg=id)case and prove its equivalence to monolithic causal attention. They then explore nonidentity transformations including KV quantization, selective retention, and positional re-addressing, as well as transferring selected KV entries between inference processes. The authors' evaluation includes next-token prediction on PG-19, needle-in-a-haystack retrieval at up to 128K tokens, and sparse KV transfer experiments where a consumer recovers a target fact using a small subset of the producer's cache.

The technical contribution seems rather limited. The proposed framework mainly provides a conceptual formulation that encompasses existing KV-cache operations, and I don't see much new algorithmic insight arising from this formulation. Specifically, the transformations essentially reuse established KV quantization (e.g., KIVI), pruning (e.g., SnapKV), and selection techniques. The sparse KV transfer results are interesting, though they seem more like empirical observations under a fairly restricted experimental setup (I'm not sure how much they tell us about the framework itself). Overall, I am not convinced that the paper establishes a sufficiently novel or practically useful contribution.

Critical strengths
  • The paper provides a unified formulation for describing different KV-cache operation types, which could be useful for organizing and studying these techniques. And, even though I do see some value in having a common abstraction for these operations, because the underlying techniques are already known the paper might not be a good fit for an ICLR publication.

  • The sparse KV transfer experiments are interesting. In particular, the observation that a consumer can potetnially recover a designated piece of information using only 36 instead of 32,768 cache positions suggests that task-relevant information can sometimes be preserved in a small subset of the KV state. This seems like one of the more interesting observations in the paper overall.

Critical weaknesses
  • I don't see much novelty in the core framework. The identity case is standard chunked prefill, while quantization, pruning, and KV transfer have already been studied in prior work. It is unclear to me what new capability the proposed formulation actually enables. I do see that there is a different perspective the paper enables for how all of these operations can collectively be looked at. But at the actual runtime efficiency level I don't see what much novelty.

  • I am not convinced that the nonidentity experiments establish the usefulness of KV-Fold itself. Findings such as int4 outperforming int3 or question-aware selection outperforming random selection seem rather expected. I also think stronger comparisons with existing methods are needed. It's fascinating to me that some of the most important existing methods (e.g., KIVI, KVQuant in the quantization domain) are only referened in the discussion section and not compared again exprementally.

  • The sparse KV transfer experiments assume access to the full cache and the consumer's question for selection. If the producer already has the context and the question, why not simply compute and transmit the answer instead of the 36 KV positions? There isn't much evidence of a practical communication advantage here.

  • The evaluation mainly relies on synthetic needle retrieval, which seems fairly limited for the broader claims. More complex reasoning tasks are missing, and the sparse transfer results are mostly limited to Llama-3.1-8B (and the Qwen replication for transfer results, as reported under "Qwen replication status." in Appendix E, does not seenm to work reliably).

  • I am not convinced by the practical efficiency claims. Quantization, specifically, is simulated while retaining bf16 storage, so the reported memory savings are not actually realized. The identity case also retains the full KV cache, and the proposed constant-memory design remains unimplemented.

  • Several conclusions rely on very small samples. Selective quantization uses only 10 trials per setting, with improvements from 3/10 to 5/10 or 6/10 that are statistically insignificant as the authors themselves mention. So how are the authors even confident in the robustness of these findings?

Questions for authors
  • What concrete capability or theoretical insight does the KV-Fold formulation provide beyond applying existing KV-cache quantization, pruning, and selection methods (during standard chunked prefill)? Could the authors give an example that specifically requires the proposed framework?

  • To my question already stated in the weaknesses section, what is the intended practical advantage of sparse KV transfer when the producer already has the full cache and evaluates the consumer's question against it? In particular, how does the total computational/communication cost compare w/ transmitting a decoded answer or even a short retrieved text span?

  • Could the authors provide more direct comparisons against existing KV-cache compression methods, including but not limited to quantization methods? (under matched memory budgets and evaluation settings)

  • How well do the sparse transfer results generalize to more complex downstream tasks, multiple questions, or questions different from the one used for selecting the KV positions?

Can the authors also demonstrate actual GPU-memory savings and end-to-end inference efficiency using compressed KV storage? Again, how would these results compare with existing optimized chunked-prefill implementations?

8 comments

We thank the reviewer for their constructive feedback. We address the main concerns below.

Novelty and Theory.

Response We agree that quantization, pruning, and chunked prefill are not new techniques, and we do not claim otherwise. Our motivation was to ask whether, rather than treating the KV cache as a passive byproduct of inference, we can treat it as a computational state that can be explicitly programmed.

This perspective was inspired by the foldl (left-fold) abstraction in functional programming, where a computation evolves by repeatedly applying an operation to an accumulated state. KV-Fold brings this idea to pretrained transformers. At each step, an operation modifies the current KV state before the frozen transformer processes the next input. Standard chunked prefill leaves the state unchanged between steps, but this is only one possible choice.

We call this perspective KV-Space Programming. It is closely related in spirit to recent work on Weight-Space Programming (Takehi et al), which explores modifying model parameters directly. Our approach instead asks how much of a model's behavior can be changed by manipulating its inference-time state, without changing its weights.

A theoretical connection. Under explicit assumptions, a programmable KV state can approximate any prescribed mapping from a finite set of distinct unit-normalized attention queries to desired attention outputs while keeping the model weights fixed. This result assumes that the state transformation can write arbitrary keys and values and that attention is restricted to those entries.

Proof sketch. For each query, we write a key pointing in the same direction and store the desired output as its value. Increasing the key magnitudes makes the matching entry receive nearly all of the attention. The weight assigned to other entries decreases exponentially, which bounds the output error. Extending the result to model outputs requires a fixed continuous readout whose range contains the desired outputs. The full construction, bound, assumptions and proofs appear in Sections 1 and 2 of the mathematical supplement.

Our sparse-transfer experiments offer an initial illustration. A recovered 20-trial reproduction over six reused PG-19 documents with 32K-token inputs achieves 20/20 correct answers with 36 retained KV positions and 20/20 with the 167-position compacted, re-addressed condition.

What makes this perspective particularly interesting is that the KV state does not have to remain tied to the computation that produced it. Our experiments study selective modification and, separately, transfer between inference processes without retraining or changing the underlying model. The recovered reproduction illustrates sparse factual access. The new experiment below tests recovery of an ongoing computation.

We believe this points to a broader opportunity than simply making KV caches smaller. Most existing compression methods ask how much information can be removed without sacrificing performance. KV-Fold lets us explore what a pretrained transformer can accomplish when its internal state becomes something we can explicitly manipulate.

Resuming an ongoing computation. QASPER tests question answering over scientific papers, with separate answer and supporting-evidence scores. In our initial study of five QASPER papers and twelve questions with Qwen3-8B, we checkpointed the KV state and required continuation metadata after 256 generated tokens and recovered all twelve continuations exactly, including generated tokens, final text, EOS and the run's random-number generator state. All logits checked after resuming generation matched exactly. Against text replay using the original processing schedule, the original three-question CrossWOZ comparison reduced receiving time by 44.17%. In three separate repeats with the methods run in different orders, the median reduction was 43.48%. The four additional papers showed 24.47–70.04% reductions, with one receiving observation per condition and paper. These clocks include restore or text prefill/replay and decoding, but exclude model loading, prior checkpoint creation and transport. The checkpoints are question-specific and distinct from reuse of a document cache built without access to the questions.

A preregistered expansion with paper-level uncertainty. We then selected fifteen additional papers from the QASPER training split and 33 questions before observing model outputs. We filtered documents using the fixed input-length and format limits, excluded the initial papers, and selected eligible papers in a fixed order computed from document hashes. We then selected questions using a separate fixed ordering of hashes of their IDs, before inspecting the question text or answers. Fourteen papers (30 questions) completed the protocol. Every checkpoint continuation matched the reference in tokens, text, EOS and the run's random-number generator state. All logits checked after resuming generation matched exactly.

Note. One registered paper reached the unchanged 2,048-token cap before its receiving comparisons. Its capped and unattempted outcomes are retained, with no replacements. The interval is conditional on the fourteen completed paper pairs.

Checkpoint receiving was faster than exact text reconstruction on all fourteen papers. We calculated the geometric mean of the checkpoint-to-replay time ratios, giving each paper the same weight. This gives a receiving-time reduction of 47.85% (95% paired-paper bootstrap interval of 40.33–55.76% from 20,000 resamples), with per-paper reductions of 22.53–77.89%. Each paper contributed one measured set of three receiving conditions on the same H100, with order balanced across the registered papers. The interval summarizes across-paper variation in this eligible cohort under an exchangeability assumption, not run-to-run timing uncertainty. The original five-paper pilot is excluded. Questions and repeated arms are not treated as independent papers.

Exact text replay rebuilds the saved state by processing the document in 512-token chunks, followed by the question and then the previously generated tokens one at a time. This follows the original run's processing schedule. Fast replay instead processes the question and generated text together in 512-token chunks, which can change the continuation.

Restoring the checkpoint avoids this replay, reducing receiving time and its estimated cost while preserving exactly the same continuation as exact replay. On the fourteen completed new papers, both methods scored 0.446 for answer F1 and 0.307 for evidence F1. We calculated each average by first averaging question scores within each paper, then averaging across papers. The appendix reports the fast-replay baseline's timing and answer quality.

This is where the functional formulation and our expressivity result become significant. Together, they provide a principled way to reason about state transformations, while our experiments distinguish sparse factual access from exact recovery of an ongoing computation. Rather than viewing the KV cache solely as a memory optimization problem, we see it as an additional interface through which the behavior of a frozen transformer can be controlled.

Practical Motivation for Sparse KV Transfer

Response For a single factual question, transmitting the decoded answer or a short text passage is certainly more efficient, and we do not intend to suggest otherwise. Our motivation is to transfer an unfinished computation. Beyond communicating a final answer, can we transfer part of a model's internal state so that another inference process can make use of information it has never directly encountered?

The recovered sparse reproduction above records factual access using 36 of 32,768 positions. Our new experiment separately tests transfer of the complete KV cache to receiving processes that never read the original document. They answer multiple questions without reconstructing the document cache for each question.

What we find particularly interesting is the possibility of sharing representations that another instance of the same model can directly attend to and continue reasoning over. This could be useful in settings involving multiple agents, different downstream objectives, or continued inference over shared information. In particular, branching work-trees where multiple independent parallel agents must operate on the same originating state (a very common scenario in modern agentic pipelines) are useful extensions of this method.

The checkpoint result avoids sequential reconstruction while preserving the same continuing computation. At the recorded rental rate of USD 2.19 per GPU-hour, the estimated cost of the receiving computation in the three-question CrossWOZ comparison was USD 0.01959 for exact text replay versus USD 0.01093 for checkpoint restoration, a USD 0.00865 (44.17%) reduction. We also verified this capability between two physical H100 GPUs. Starting from the 256-token checkpoint, the receiver generated the remaining 1,247 tokens exactly, yielding the same complete 1,503-token sequence, final text, EOS and the run's random-number generator state. For each of the remaining 1,247 tokens, all 151,936 logits checked after resuming generation matched exactly. This demonstrates transfer of an ongoing computation between separate processes. A decoded answer communicates a result, while a checkpoint lets the receiver continue the same unfinished computation. The measured saving concerns loaded-model receiving time. Complete two-host answer-delivery costs, including writing and transport, are reported separately in the appendix.

Comments on the paper

1 comment
Area_Chair_RmRLOpenReview ↗

Discussion Phase

As the discussion period is rapidly coming to a close (Nov 18 AoE), I would like to remind the reviewers to please participate in discussion and acknowledge any comments posted by the authors. A secondary, important reminder: full disclosure of any AI usage is required and the reviewer must include their own human-written assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.