acceptodds
Under review as a conference paper at ICLR 2027

DiaKV: Offline KV Cache Training for Context-Level Adaptation

Abstract

Test-time training is a learning paradigm that updates the weights of the model on a test-time input, enabling more granular adaptation than fine-tuning. However, adaptation performed after a request arrives adds online training latency, limiting its use in latency-sensitive serving. We propose DiaKV, which trains the KV cache of a reused context offline before future queries arrive. DiaKV directly optimizes the KV cache of a context while keeping the model weights and cache length fixed. We formulate this adaptation as KV-cache posterior maximization given context-specific question-answer distribution, and augment 20 seed QAs per context into a larger training set. Across four long-context QA benchmarks and three 3–4B LLMs, DiaKV improves the primary macro metric in 11 of 12 model–benchmark pairs.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM, LLM adaptation, KV cache, test-time training

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_5Q4pOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes DiaKV, which optimizes the cached context representation through supervised learning. Unlike traditional test-time training, which introduces additional latency by updating model parameters after a query arrives, DiaKV performs adaptation offline for contexts expected to be reused. Specifically, the method augments 20 seed question-answer pairs into 800 training examples and optimizes the corresponding KV cache through supervised learning.The results suggest that offline KV-cache optimization can improve context-specific QA performance without introducing additional online training latency.

Critical strengths
  • The paper explores a promising direction by treating the KV cache as a trainable representation. This provides an interesting alternative to conventional weight-based fine-tuning and test-time training.
  • The method achieves consistent improvements across multiple long-context QA benchmarks and language models. Although its practical applicability may be limited by the offline training overhead, the results demonstrate the potential of directly optimizing KV caches for context-specific adaptation.
Critical weaknesses
  • Although DiaKV improves long-context QA performance, it remains unclear whether these gains primarily come from acquiring task-specific knowledge, improving access to existing knowledge, or preserving the model's instruction-following capabilities. For example, on LongHealth-M, LoRA and full fine-tuning perform substantially worse than the untrained base model, partly due to response-format mismatches. This raises concerns about whether DiaKV's advantage over weight-based adaptation reflects better knowledge utilization or simply greater robustness to instruction-following degradation. Additional controlled experiments would help disentangle these factors.

  • DiaKV avoids online training latency by shifting adaptation to an offline stage, but still requires substantial computation to generate synthetic QAs and optimize the KV cache for each context. This limits its applicability to newly arriving or frequently updated documents, where repeated optimization may be costly. Although the method demonstrates the benefits of KV-cache optimization, the paper does not sufficiently analyze when the accuracy improvements justify the additional preparation cost, particularly under varying context reuse frequencies and update rates.

  • It remains unclear how well the optimized KV cache generalizes to queries that substantially differ from the training distribution. Although paper investigates the effects of seed QA coverage, it does not fully address generalization to unseen question types or reasoning patterns.

Questions for authors
  • Could the authors provide experiments that distinguish improvements in knowledge utilization from the preservation of instruction-following behavior, such as evaluating answer correctness independently of output-format compliance or using format-aligned training data?

  • Could the authors evaluate DiaKV on out-of-distribution queries, such as unseen question types or multi-hop reasoning tasks, to better understand its generalization beyond the augmented QA distribution?

4 comments
AuthorsOpenReview ↗

Author response (1/3)

We thank the reviewer for your valuable feedback. We group your reviews into three buckets based on their content.

  • [W1 & Q1] Disentangling the accuracy result of DiaKV vs weight tuning
    • Rescoring LH-M with LLM-as-judge, which ignores output format, shows that weight tuning's collapse is mostly a format failure, yet DiaKV stays on par with weight tuning (ahead in 8 of 9 comparisons), so our claim of being as effective as weight tuning while strictly better for serving stands.
  • [W2] Amortization analysis
    • DiaKV targets contexts that keep hitting the prefix cache rather than frequently updated ones, and an illustrative amortization analysis shows that the preparation cost is recovered only under heavy reuse, so we scope our claim to higher accuracy without online latency, not lower cost.
  • [W3 & Q2] Out-of-distribution, multi-hop queries
    • On multi-hop rewrites of LH-M questions, a reasoning pattern absent from the training QAs, DiaKV's gain (+3.1%) is at least as large as on the original single-hop questions (+2.1%).

[W1 & Q1] Disentangling the accuracy result of DiaKV vs weight tuning

Thank you for raising an important point in our evaluation. Even though the degradation of instruction-following capability in weight tuning (§5.2) is a valuable analysis, it may conflate the benefit DiaKV brings. Therefore, following your suggestion, we evaluate the LH-M (LongHealth-M) result on answer correctness alone, independent of output-format compliance. We have tried multiple rule-based metrics, but none of them fully isolates format compliance (examples: 07/13/2019 vs 07.13.19, Right ventricular anterior (RIVA) region vs RIVA area). We use LLM-as-judge instead, with the same setup as our two other datasets, QASA-M and FinTextQA (§4.1).

Results are in the table below. Losing instruction-following capability in weight tuning explains nearly all of the severe accuracy degradation in LH-M. However, even for LLM-as-judge, which does not penalize format at all, DiaKV remains on par with weight tuning. DiaKV outperforms all 3 weight baselines (full fine-tuning, LoRA r=32, LoRA r=8) in Qwen3 and Ministral, while only falling behind LoRA r=32 in Llama by some margin.

Table R3-1. Format-free (LLM-as-judge) evaluation on LH-M

ModelParametrizationAccuracyLLM-as-judge (newly added)
Qwen3-4B-2507baseline0.7170.728±.000
KV0.7280.742±.000
full0.3360.658±.001
LoRA (r=32)0.5220.731±.000
LoRA (r=8)0.2250.711±.000
Llama-3.2-3Bbaseline0.4640.529±.003
KV0.4970.564±.002
full0.2780.520±.002
LoRA (r=32)0.2890.605±.002
LoRA (r=8)0.3310.533±.002
Ministral-3-3Bbaseline0.6690.740±.002
KV0.6920.742±.000
full0.5640.731±.001
LoRA (r=32)0.4580.701±.003
LoRA (r=8)0.6170.702±.002

Overall, format-free evaluation on LH-M shows that even factoring out the instruction-following failure, DiaKV remains competitive compared to weight tuning baselines. For clarity, we reiterate our claim in the paper regarding DiaKV vs weight tuning (§5.2): DiaKV is strictly better than weight tuning in terms of serving, while being “as effective as training weights” (L365). This format-free evaluation result does not alter our claim.

AuthorsOpenReview ↗

Author response (2/3)

[W2] Amortization analysis

Newly arriving and frequently updated context

"This limits its applicability to newly arriving or frequently updated documents, where repeated optimization may be costly."

Our motivation for DiaKV starts from a common characteristic of modern LLM serving systems — very high cache hit rate (often >90%; [1]) and increasing cache TTL (§2 - prefix caching & §4.3). We agree with your point that repeated optimization makes DiaKV costly to apply to newly arriving or frequently updated contexts. We want to clarify that such contexts incurring cache miss are not our primary targets. High cache hit rates of LLM serving systems suggest that DiaKV’s gain from cache hit contexts alone can still deliver a meaningful win for the entire system. Future work may study how to reuse DiaKV for a frequently updated context where the update is small and local. We believe such study deserves a separate work, and our work provides a groundwork for such investigation.

Amortization analysis

.. the paper does not sufficiently analyze when the accuracy improvements justify the additional preparation cost, particularly under varying context reuse frequencies and update rates.

The general formula for cost amortization is as follows, assuming the same accuracy with and without DiaKV:

α⋅FC+N×MC<N×C  ⟺  N>α⋅FCC−MC\alpha \cdot FC + N \times MC < N \times C \iff N > \frac{\alpha \cdot FC}{C - MC}

  • N = #(cache hit)
  • α\alpha is the discount factor for offline compute
  • FC (Fixed Cost) = preparation cost for a single context (single DiaKV)
  • MC (Marginal Cost) = per-query cost, serving with DiaKV
  • C (Cost) = per-query cost, serving without DiaKV (i.e., a larger model matching DiaKV's accuracy)

The discount factor α\alpha defines how much we discount for the offline resource. Focusing only on latency means α\alpha = 0, as there is no “offline latency”. For a resource-constrained environment where offline resource use is as costly as online use, α\alpha will be 1 or nearly 1.

It is hard to translate accuracy gain into cost measures like dollars or FLOPs. Our approach equates DiaKV’s accuracy gain from x to y with a cost gain: the cost of running a larger model whose accuracy is y minus the cost of running the target model whose accuracy is x. Note that the online cost of running target model with DiaKV is exactly identical to the online cost of running target model alone.

We use Llama-3.2-3B as an example target model. For preparation cost, we use USD 3.4 per context, which is the median over 4 datasets (Appendix C.3).

  • FC = USD 3.4 (Appendix C.3)
  • C - MC = USD 0.001 × (2.7 - 1) = USD 0.0017
    • Llama-3.2-3B with DiaKV reaches 97.6% of the untrained Llama-3.1-8B's score in geometric mean over the four benchmarks.
    • Assume running a single query on Llama-3.2-3B costs 0.1 cents and the cost gap between the two is proportional to the model size gap (2.7x larger)

Under this illustrative assumption, with α = 1, the preparation cost is amortized after approximately 2,000 cache hits. This is a very crude analysis, but we don’t believe the conclusion would easily change (it is an unfavorable conclusion for us, after all) — cost amortization requires heavy reuse in DiaKV.

As stated in the footnote on page 5, we have not claimed lower total cost in any part of the paper. Nevertheless, it would’ve been better to include a cost amortization analysis and honestly state that cost amortization requires heavy reuse in DiaKV.

DiaKV’s primary benefit is improved answer quality without additional online training latency, rather than lower total cost. This may be suitable for some (but not all) of the real-world scenarios where resources are relatively ample (so additional offline preparation cost is affordable) but maximizing accuracy and minimizing latency still matter. Prior works like sleep-time compute [2] or offline recurrence [3] also mostly leave out offline cost and prioritize online latency and accuracy. We will state our claim scope more clearly and report the amortization calculation only as an illustrative example.


[1] TraceLab: Characterizing Coding Agent Workloads for LLM Serving (Zhu et al., arXiv 2026)

[2] Sleep-time Compute: Beyond Inference Scaling at Test-time (Lin et al., arXiv 2025)

[3] Language Models Need Sleep (Lee et al., SPIGM Workshop @ ICML 2026)

AuthorsOpenReview ↗

Author response (3/3)

[W3 & Q2] Out-of-distribution, multi-hop queries

As the review recognizes, our §6.2 analysis addresses seed bias from locality, but does not fully address seed bias concern from other factors, including the factors the reviewer mentions — question types and reasoning patterns, e.g., multi-hop reasoning.

To address this weakness & question, we set up a controlled experiment on whether DiaKV’s gain extends to multi-hop queries, which is out-of-distribution to the training questions.

Our study is done on LH-M (LongHealth-M), one of the datasets we used in the paper (§4.1). A single LH-M context contains 10 patient cases, each including the patient name and his/her clinical documents. Most of the training questions (1957 of 1960) and the test questions (339 of 360) directly ask about a patient, referring to them by name. We consider such questions as single-hop questions. For these 339 single-hop test questions, we create multi-hop test questions by replacing the patient name with different facts about the patient. For example, if the original single-hop test question is:

When was the total spine MRI plain conducted in Laura Miller?

then the constructed multi-hop question is:

Consider the patient with hepatocellular carcinoma and liver cirrhosis. When was the total spine MRI plain conducted in this patient?

These questions are multi-hop as they require (1) identifying the patient among 10 given patients (who has hepatocellular carcinoma and liver cirrhosis? → Laura Miller) and then (2) locating the fact about that patient (When was the total spine MRI plain conducted in Laura Miller? → 2020-04-06). The answer remains identical to the single-hop question.

On the three models we’ve used in the paper (§4.1), we evaluate the same DiaKV used in the paper, which are trained on mostly single-hop questions. The results on original single-hop test questions and newly added multi-hop questions are in Table R3-2. Across three models, the mean accuracy gain in multi-hop questions is +3.1%, even higher than that of single-hop questions, +2.1%. The trends are similar for an auxiliary metric, word-level F1. This suggests that DiaKV’s gains transfer to reasoning patterns unseen in the training set, multi-hop reasoning in this case.

Table R3-2. LH-M results on the 339 multi-hop (out-of-distribution; OOD) vs single-hop questions (in-distribution; ID)

ModelQuestionsAcc. untrainedAcc. trainedF1 untrainedF1 trained
Qwen3-4B-2507single-hop (ID, original)0.7080.729 (+3.0%)0.8020.825 (+2.9%)
multi-hop (OOD)0.6960.708 (+1.7%)0.7860.813 (+3.4%)
Llama-3.2-3Bsingle-hop0.4780.487 (+1.9%)0.6020.635 (+5.5%)
multi-hop0.4450.490 (+10.1%)0.5840.639 (+9.4%)
Ministral-3-3Bsingle-hop0.6670.678 (+1.6%)0.7900.809 (+2.4%)
multi-hop0.6870.690 (+0.4%)0.7940.809 (+1.9%)
Mean of 3single-hop0.6180.631 (+2.1%)0.7320.756 (+3.3%)
multi-hop0.6100.629 (+3.1%)0.7210.754 (+4.6%)

Note that the accuracy and F1 for single-hop questions may slightly differ from those reported in the paper, as these results are on the 339-question subset (out of 360 in total) for comparison.

Thank you for your detailed response and additional experiments. These have addressed several of my concerns. I have therefore decided to raise my score to 3.

Reviewer_H1ArOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes DiaKV, which trains the KV cache of a reused context offline, positioning context-level adaptation between model-level fine-tuning and context-level adaptation. Training of DiaKV is defined as MAP estimation of the KV given a context-specific QA distribution. Across 4 long-context QA and 3 models, DiaKV improves performance in most cases. Also, It generally outperforms textual context optimization (ACE) and qTTT's self-supervised loss, and matches parameter updating methods.

Critical strengths
  • The motivation is clear. Full-model tuning is too expensive, while the serving granularity of per-request TTT is too fine. Since serving systems largely reuse the KV cache, this adaptation targets the middle ground between the two.
  • The design is serving-friendly. It does not require any additional tokens or adapters at inference time. Serving costs are also reported (e.g., TTFT, LoRA overhead).
Critical weaknesses
  • The proposed method’s practical applicability is limited.
    • The trained KV caches are tied to exact prefix tokens and positions. The method is therefore only applicable when exactly the same text is repeated across requests, which is not very realistic. In RAG, for example, the order of documents can be interleaved, or some documents can be missing depending on the given query. The proposed method cannot be applied in these cases.
    • The method requires 20 question-answer pairs per context, labeled by models stronger than the target model. Its applicability is unclear when labeled data is unavailable or when no stronger model exists.
  • The evaluation does not cover the target use case or include proper baselines.
    • The evaluation targets only long-context QA. To motivate the method’s practical use, the evaluation should consider realistic prefix-cache reuse scenarios or query drift over the same contexts.
    • Prefix-style baselines are not included, e.g., Prefix-Tuning (ACL’21), Cartridges (ICLR’26), and GradMem (ICLR’26).
  • The method requires more elaboration.
    • Its positioning between fine-tuning and TTT is quite similar to that of Context Tuning (ICML’26), and the method also shares some commonalities with it.
    • The use of a stronger model for labeling needs more elaboration, since other TTT methods do not require explicit labels.
Questions for authors
  • How do you differ from Context Tuning?
  • In deployment, where would the QA pairs come from? If the contexts come from a retrieval corpus or differ largely across queries, how does DiaKV handle these scenarios?
  • How does the method scale to models larger than 4B? How effective is it for recent models that use latent KV or native sparse attention?
11 comments
AuthorsOpenReview ↗

Author response (1/9)

We thank the reviewer for your review. We refer to each of your review based on the order written (top to bottom); for weaknesses, the numberings are based on 2nd-level bullets.

We group your reviews (6 weaknesses and 3 questions) into 8 buckets based on their content.

  • [W1 & 2nd part of Q2] DiaKV for position-independent context caching (for RAG)
    • We scope DiaKV to ordinary prefix-cache reuse, where each composed context is one fixed prefix; a feasibility study linking separately trained DiaKVs with EPIC's LegoLink-0 recovers part of the linking loss on single-hop questions but gives mixed results on cross-context questions, which we leave to future work.
  • [W2] Augmentation when labeled data is unavailable or when no stronger model exists.
    • Even with small models (3–4B active parameters) generating their own training QAs, self-augmented DiaKV still improves 4 of 7 model–benchmark settings without notable degradation, so a stronger model makes gains more consistent but is not strictly required.
  • [W3] More realistic prefix-cache reuse scenarios
    • On a growing chat history (HaluMem), a more realistic prefix-reuse scenario, incrementally trained DiaKV improves Llama and Ministral but not Qwen3, a preliminary result that leaves room for further study.
  • [W4] Comparison to prefix-style baselines
    • DiaKV is itself a prefix-style method that, unlike Cartridges and GradMem, aims to exceed rather than compress the full context, and prefix tuning on QASA-M falls short of DiaKV on Llama and Ministral and only matches it on Qwen3.
  • [W5 & Q1] Comparison to context tuning
    • Context Tuning trains the KV of few-shot demonstrations at inference time, whereas DiaKV adapts general long contexts, which supply no input–output pairs, using augmented QAs and trains offline before queries arrive.
  • [W6] Elaboration on the necessity of labels
    • DiaKV needs a small set of labels because it sits between label-free TTT and label-heavy fine-tuning; our ablations show that seed QAs make gains more consistent than a self-supervised loss or seedless augmentation, and external supervision is common in TTT as well.
  • [1st part of Q2] Where would QAs come from in deployment
    • Seed QAs are application-specific and arise naturally in settings such as long-lived rule documents with Q&A pages, or agent tool and skill definitions that include examples.
  • [Q3] Larger / different attention variant models
    • DiaKV carries over to latent (MLA) and native sparse attention and to 24–69B models, improving QASA-M by +4.1% to +10.1% (macro) on Mistral-Small-3.2-24B, GLM-4.7-Flash, and LongCat-Flash-Lite-Sparse.

[W1 & 2nd part of Q2] DiaKV for position-independent context caching (for RAG)

On the scope of DiaKV, and a note on our composed contexts

Thank you for raising this. You are right that a trained DiaKV, as presented in the paper, is tied to its exact prefix: it can be reused only where an ordinary prefix cache would be hit. We scoped the paper to this setting deliberately. As you note among the strengths ("serving systems largely reuse the KV cache"), prefix-cache hit ratios are high in LLM serving systems [1], and cache retention keeps growing (§2, §4.3), so a long-lived prefix cache was our starting point.

One point we should have stated more explicitly in §4.1 concerns the composed contexts, QASA-M and LH-M. We composed multiple papers or patient cases into one context for an operational reason: a single paper or patient case is only 4-5k words long and has too few QAs to supply both seed and test sets, which gives DiaKV little to train on. Each QASA-M or LH-M context is therefore treated as one fixed prefix, following Cartridges [2]; the paper does not train separate per-document caches or combine them at inference. We will make this explicit in §4.1 of the revision.

That said, the setting you describe, where documents are reordered or dropped per query as in position-independent caching (CacheBlend, EPIC) [3, 4], is a natural and interesting extension: can separately trained DiaKVs be composed at inference time? We ran feasibility experiments on this, reported below.


[1] TraceLab: Characterizing Coding Agent Workloads for LLM Serving (Zhu et al., arXiv 2026)

[2] Cartridges: Lightweight and general-purpose long context representations via self-study (Eyuboglu et al., ICLR 2026)

[3] CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion (Yao et al., EuroSys 2025)

[4] EPIC: Efficient Position-Independent Caching for Serving Large Language Models (Hu et al., ICML 2025)

AuthorsOpenReview ↗

Author response (2/9)

[W1 & 2nd part of Q2] DiaKV for position-independent context caching (for RAG) (continued)

Experiment on composing DiaKV”s”

Our headline is: DiaKV can be extended for position-independent context caching by integrating methods like LegoLink-0 from EPIC [4], but its empirical success requires more validation.

We experiment integrating DiaKV with LegoLink-0, an overhead-free KV linking algorithm. LegoLink-0 simply concatenates separately prefilled KV caches while dropping the first few tokens of each cache (except for the first cache). This naturally aligns with DiaKV, as DiaKV freezes the prepended system prompt during training. For DiaKV + LegoLink-0, we drop the system prompt of Nth context (N>1), which was prefilled but not updated during DiaKV training (so no trained parameters are lost from linking).

For the experiments, we compose QASA-M contexts (total 9) No. 3+8 and 4+7 (365 of all 1,027 test questions) because two contexts must fit together in Llama-3.2-3B's 128K-token window. In addition to the original test questions which can be answered by a single context (single-hop), we newly introduce questions which can only be answered by both contexts (multi-hop). The original single-hop questions aim to measure degradation from KV linking, while the new multi-hop questions aim to measure whether information from the linked KVs are composable.

Such “multi-hop” evaluation typically require dedicated datasets [5], but to reuse QASA-M dataset, we adopt a simpler approach — for two papers each drawn from a different context (a context contain multiple papers; §4.1), ask which was published earlier. The concrete question is:

Which of these two papers was published earlier: <title A> or <title B>? Answer with the paper's title.

This approach has multiple benefits: (a) it requires compositional reasoning as the papers do not carry explicit dates and the order must be inferred from their content, (b) a definite answer can be derived by comparing arXiv identifiers included in QASA, and (c) it is easy to scale up the number of questions by considering all combinations of papers.

Across 2 context pairs, this results in total 103 questions. We evaluate using LLM-as-judge (gpt-5.1-2025-11-13, reasoning effort = none) against the correct title, averaged over two different orders of contexts. Each question lists the two titles in one random order (the correct title first in 50 of 103).


[4] EPIC: Efficient Position-Independent Caching for Serving Large Language Models (Hu et al., ICML 2025)

[5] ♫ MuSiQue: Multihop Questions via Single-hop Question Composition (Trivedi et al., TACL 2022)

AuthorsOpenReview ↗

Author response (3/9)

[W1 & 2nd part of Q2] DiaKV for position-independent context caching (for RAG) (continued)

Experiment on composing DiaKV”s” (continued)

Considered baselines respectively for single-hop questions (the original QASA questions) and multi-hop questions (which is published earlier?) are:

  • Single-hop (answered by context A, context B is irrelevant)
    • A only
    • A only + DiaKV
    • A + B, joint prefill
    • A + B, linked: each cache computed alone, then linked with LegoLink-0
    • A + B, DiaKV each and linked: each cache computed and trained alone, then linked with LegoLink-0
  • Multi-hop (only answerable using both A and B)
    • ∅\emptyset (closed-book): no context given; to control for model’s parametric knowledge
    • A + B, joint prefill
    • A + B, linked
    • A + B, DiaKV each and linked

Table R2-1. QASA-M 2 context pairs, LLM-as-judge accuracy on single-hop questions

Context in cacheLlama-3.2-3BMinistral-3-3B
Asked context only0.3160.394
+DiaKV0.365 (+15.5%)0.477 (+21.1%)
Both contexts, joint prefill0.2820.382
Both contexts, linked0.2380.328
+DiaKV0.258 (+8.4%)0.402 (+22.6%)

Table R2-2. QASA-M 2 context pairs, LLM-as-judge accuracy on multi-hop questions

Context in cacheLlama-3.2-3BMinistral-3-3B
None (closed-book)0.4170.573
Both contexts, joint prefill0.5870.709
Both contexts, linked0.6650.670
+DiaKV0.553 (−16.8%)0.704 (+5.1%)

Results in Table R2-1 and R2-2 show mostly positive results with one odd case.

For single-hop QAs (Table R2-1), using both contexts result in a noticeable drop compared to the single context case (row 1 vs 3), likely due to context pollution and nearing max model context length. When linking two separately prefilled contexts (row 4), the drop further continues, as expected. Applying DiaKV to each of the context and linking them (row 5) recovers some of that drop; for Ministral it surpasses the joint prefill itself.

For multi-hop QAs (see Table R2-2), the results are mixed. Ministral-3-3B presents a positive case: linking two KVs lose 3.9%p, but DiaKV recovers this drop from linking (+3.4%p), resulting in similar accuracy to joint prefill.

However in Llama-3.2-3B, DiaKV + linking degrades from linking alone (-16.8%). Surprisingly, linking outperforms joint prefill by a large margin (+7.8%p); typically what to expect is the other way around [4], demanding a closer investigation at Llama-3.2-3B run.

From the results so far, it is not definitive whether DiaKV can be trivially integrated with position-independent context caching. We believe this problem deserves a separate paper. Future works can also extend on linking algorithm with overhead (LegoLink-k (k>0) or CacheBlend) or testing more compositional queries. Our work DiaKV provides a valuable groundwork for such future extensions.


[4] EPIC: Efficient Position-Independent Caching for Serving Large Language Models (Hu et al., ICML 2025)

Reviewer_ykvcOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

DiaKV adapts an LLM to a reusable context by optimizing its prefilled KV cache offline, while keeping the model weights and cache length fixed. It expands 20 context-specific seed QA pairs into approximately 800 training examples using separate question and answer generators, then optimizes the cache on question and answer tokens. The objective is motivated through a Bayesian/MAP formulation. Evaluation covers four long-context QA benchmarks, 43 contexts, and three 3-4B models, with improvements in 11 of 12 primary macro model-benchmark comparisons. Comparisons with textual context optimization, weight tuning, and a qTTT-based objective, together with seed-count and locality analyses, investigate the benefits and limitations of the recipe.

Critical strengths
  1. Substantial empirical gains across several models and domains. The evaluation covers 43 contexts, four QA benchmarks, and three model families, with improvements in 11 of 12 primary macro comparisons. NarrativeQA shows particularly substantial gains: macro ROUGE-L increases from 0.167 to 0.301, 0.265 to 0.317, and 0.206 to 0.314. These results are the strongest evidence supporting acceptance: the proposed recipe produces useful improvements across multiple evaluated settings, with benefits large enough in some cases to be practically meaningful.

  2. A practical approach to improving reusable contexts. Training is performed offline while model weights and cache length remain fixed. Under the assumed prefix-cache reuse setting, the trained cache can improve answers without online gradient updates or additional attention positions. This is an appealing tradeoff for workloads that repeatedly query the same documents and can amortize preparation cost.

  3. Informative comparisons with alternative adaptation strategies. The same-data weight-tuning experiments and approximately training-token-matched qTTT-KV comparison investigate both the update location and training signal. The LH-M analysis is especially useful: it identifies an augmentation-format mismatch and shows that cache tuning preserves instruction following better under that mismatch, providing insight beyond aggregate benchmark scores.

  4. Direct examination of the role and coverage of task supervision. The seed-count and document hold-out studies test a central assumption of the method. Their mixed results help identify when seed QAs matter and what happens to documents not covered by those seeds, giving the empirical findings a useful account of their limitations.

Critical weaknesses
  1. The incremental benefit over the closest cache-training recipe is not fully established. Cartridges already trains corpus-specific KV caches offline and evaluates cross-entropy training on synthetic conversations. DiaKV's distinct contribution therefore rests mainly on its full-length, seed-informed recipe and empirical analysis. The existing seed-count ablation supports the value of task information, but does not fully establish how the combined changes in supervision and data generation improve upon generic self-study. This leaves the methodological advance less clearly established than the usefulness of the complete recipe, and is my main reservation. The claim of introducing context-level adaptation itself should consequently be narrowed.

  2. The smaller gains have limited evidence of stability. LH-M and FinText each contain only two adapted contexts, and the main judge-based error bars measure judging noise rather than variation across adaptation runs. Appendix D.2 provides useful uncertainty analysis for the locality study, but does not establish the stability of the smaller main-table gains. This limits confidence in generalization across contexts, especially where the observed improvements are modest.

Questions for authors

Please specify how seed and test questions are separated and whether the seed-count variants share the same held-out evaluation set. For the smaller LH-M and FinText gains, what existing per-context or repeated-run evidence supports their consistency beyond the reported judging uncertainty?

3 comments
AuthorsOpenReview ↗

Author response (1/2)

We thank the reviewer for your reviews. We group your reviews into 3 buckets and address each.

  • [W1] Incremental benefit over Cartridges
    • We narrow our claim: training a KV cache is not new in itself (Cartridges, Context Tuning), but it adapts a model to general long contexts beyond compression and few-shot demonstrations, and seed QAs beat a seedless, self-study-like proxy in 9 of 12 cases.
  • [W2 & Q2] Stability of the LH-M and FinText gains
    • Three more training runs give a positive mean gain in all six settings, with run-to-run variation well below the gain in four and comparable to it in two.
  • [Q1] Seed/test separation and the seed-count evaluation set
    • Seed and test QAs are disjoint, deterministic splits of each context's original QAs, and every seed-count variant is scored on the same 1,754 held-out questions.

[W1] Incremental benefit over Cartridges

We agree that the claim should be narrowed. Cartridges [1] already trains a corpus-specific KV cache offline, and Context Tuning [2] trains a KV cache for few-shot tasks. Our claim is that DiaKV shows context-level adaptation works for general QA contexts, extending beyond the context compression of Cartridges and the few-shot setting of Context Tuning. §2 compares DiaKV with Cartridges, but that comparison was not enough. We will elaborate on it in §2 and the introduction, and add Context Tuning as a comparison (please see our response to Reviewer H1Ar, W5 and Q1).

Compression is not adaptation. A shared mechanism does not settle novelty when the goal changes. Born-Again Networks [3] reused knowledge distillation, introduced for compression, but trained a student of the same size as its teacher and showed that it outperforms the teacher. Similarly, Coprocessor [4] is "inspired by the literature on kv-cache compression", yet states that "rather than focusing on compression, we aim to augment the kv-cache" to improve reasoning. DiaKV makes the same move. Cartridges aims to match the full context with less memory, whereas DiaKV keeps the full-length cache and aims to exceed it.

What the new goal requires. To exceed the full context, training needs information about how the context will be queried, which DiaKV takes from seed QAs. A self-supervised loss on the context alone stays below the baseline or within 0.015 of it in 11 of 12 cases (§5.3). The nearest proxy we have for generic self-study is the 0-seed setting of §6.1, which generates QAs from the context without any task information. It is only a proxy: its QAs are written by external models and it is trained with cross-entropy, whereas Cartridges generates its data with the adapted model and distills from it. Table R1-1 compares it with 20 seed QAs on every benchmark. Seed QAs help in 9 of 12 cases, including every model on NQA (+0.033 to +0.045). The three exceptions are QASA-M with Qwen3-4B-2507 (−0.001), LH-M with Ministral-3-3B (−0.016), and FinText with Qwen3-4B-2507 (−0.028); on LH-M and FinText, differences of this size are comparable to the variation between repeated training runs (see W1.2). Seed QAs also bring a risk that Cartridges does not face, a bias toward the seeded parts of the context, which we examine in §6.2 and in our response to Reviewer 5Q4p (W3 and Q2).

Table R1-1. DiaKV without and with seed QAs, on the prime metric of each benchmark (macro; ROUGE-L for NQA, LLM-as-judge for QASA-M and FinText, accuracy for LH-M). On QASA-M, the answers of the 0-seed setting were generated with a lower verbosity setting of the answer model.

BenchmarkModelBaseline0 seed QAs20 seed QAs (DiaKV)20 − 0
NQAQwen3-4B-25070.1670.2630.301+0.038
Llama-3.2-3B0.2650.2720.317+0.045
Ministral-3-3B0.2060.2810.314+0.033
QASA-MQwen3-4B-25070.4740.4610.460−0.001
Llama-3.2-3B0.3450.3710.404+0.033
Ministral-3-3B0.4000.4580.467+0.009
LH-MQwen3-4B-25070.7170.6940.728+0.034
Llama-3.2-3B0.4640.4530.497+0.044
Ministral-3-3B0.6690.7080.692−0.016
FinTextQwen3-4B-25070.3250.4040.376−0.028
Llama-3.2-3B0.3210.2310.377+0.146
Ministral-3-3B0.3270.2870.341+0.054

Other contributions.

  • Serving. Weights and cache length are unchanged, so the trained cache is served exactly like a plain prefix cache, with no online training, and its preparation fits within the lifetime of the cache (§4.3).
  • Formulation. We cast context-level adaptation as maximum a posteriori estimation of the KV cache under a context-specific QA distribution (§3.1). The objective consists of a prior on the KV cache, the question likelihood, and the answer likelihood, which is why DiaKV trains on question tokens as well as answer tokens, with weight decay implementing the prior.
AuthorsOpenReview ↗

Author response (2/2)

[W2 and Q2] Stability of the LH-M and FinText gains

We agree that the ± in Table 2 covers judge noise only. To measure the variation across adaptation runs, we repeated the training of the reported configuration on the same data: three more runs for every model on FinText and LH-M. Table R1-2 gives the mean and the standard deviation of the gain over these runs.

Across the four runs of each configuration, the mean gain over the untrained cache is positive in all six LH-M and FinText settings. For four of them (LH-M Qwen3 and Ministral, FinText Qwen3 and Llama), the standard deviation across runs is well below the mean gain. For LH-M Llama-3.2-3B and FinText Ministral-3-3B, the standard deviation is comparable to the mean, so those two gains should be read as positive on average but not individually reliable at this sample size. The runs reported in Table 2 lie above the four-run mean in five of six setting.

Table R1-2. Gain of the final checkpoint over the untrained cache on the prime metric (macro), as mean and standard deviation over the runs of one configuration: the reported run and its repeats on the same data. Training is not bit-reproducible, so the runs differ.

BenchmarkModelRunsMean gainStd. across runs
LH-MQwen3-4B-25074+0.0100.002
Llama-3.2-3B4+0.0220.016
Ministral-3-3B4+0.0160.006
FinTextQwen3-4B-25074+0.0340.011
Llama-3.2-3B4+0.0380.015
Ministral-3-3B4+0.0120.011

[Q1] Separation of seed and test questions, and the evaluation set of the seed-count variants

Seed and test questions are disjoint parts of each context's original QAs, fixed before any stage of the pipeline, and every seed-count variant is evaluated on the same held-out set.

Separation. For each context, we split the benchmark's own QAs once: 20 become the seed QAs and all the others form the test set. Over the 43 contexts this gives 860 seed QAs and 1,754 test QAs (292 in NQA, 1,027 in QASA-M, 360 in LH-M, and 75 in FinText). The split is a deterministic rule on the order of the dataset and involves no model output, so it is identical for every model and run:

  • NQA and FinText: the first 20 QAs of the context.
  • QASA-M: round-robin over the member papers, so that every paper contributes at least one seed QA.
  • LH-M: two QAs per patient record, chosen with a fixed random seed.

Test QAs never enter the pipeline, including augmentation and training.

The default seed QAs drawn in QASA-M and LH-M are evenly distributed across the context. We challenge this setup in §6.2 by drawing seed QAs from only certain papers or patient records, and "the results show weak evidence of bias from seed locality" (L471).

Seed-count variants. Every seed-count variant is evaluated on the same held-out set. The 5-seed variant uses the first 5 of the 20 seed QAs and the 0-seed variant uses no benchmark QA at all. In both, the unused seed QAs stay excluded from evaluation instead of returning to the test set. The untrained baseline and the 20-, 5-, and 0-seed variants in Fig. 4 and Table 10 are therefore scored on the identical 1,754 questions.


[1] Cartridges: Lightweight and general-purpose long context representations via self-study (Eyuboglu et al., ICLR 2026)

[2] Context Tuning for In-Context Optimization (Lu et al., ICML 2026)

[3] Born Again Neural Networks (Furlanello et al., ICML 2018)

[4] Deliberation in Latent Space via Differentiable Cache Augmentation (Liu et al., ICML 2025)

Thank you for your detailed response. I therefore maintain my positive score.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Revision update and rebuttal summary

We thank all reviewers for their constructive feedback.

We have uploaded a revised PDF. This revision covers changes to the main text only; added or revised text is in blue, and removed text is crossed out in red.

  • §1, §2: Narrowed claim vs. Cartridges; added Context Tuning and GradMem. (ykvc, H1Ar)
  • §4.1: Composed contexts are each one fixed prefix. (H1Ar)
  • §4.3: Scope of preparation cost: accuracy without online latency, not lower cost. (5Q4p)
  • §5.2: Format-free LH-M evaluation. (5Q4p)
  • §6.3 (new): Multi-hop (OOD) queries. (5Q4p)

The revision is provided for convenience only, and reading it is entirely optional. Every response below is self-contained and does not rely on the revised PDF.

Also for easier reading, we have added a summary at the beginning of our response to each reviewer. The content itself has not been changed.

AuthorsOpenReview ↗

Rebuttal highlights

Dear reviewers,

Thank you again for your time and feedback. As the discussion period is closing, we would like to gently follow up on our responses.

Below are the points that we believe matter most for the assessment; each table ID links to the response that contains it.

New experiments

  • Extended prefix-cache reuse scenarios. The paper is scoped to ordinary prefix-cache reuse, but our first results on the two extensions raised in the reviews are encouraging, though mixed in both.

    • Linking separately trained caches, as in RAG where the documents change with the query. DiaKV keeps its gain on the original questions on both models (+8.4% and +22.6% over linking untrained caches; Table R2-1). On new questions that need both linked contexts, it helps one of the two models (Table R2-2).
    • Incrementally training the cache of a growing chat history. Two of three models improve (Table R2-4).

    The mixed cases show that these extensions are not trivial. Still, DiaKV is not inherently limited to one fixed prefix, and these results open a concrete path beyond it.

  • Out-of-distribution queries. On multi-hop rewrites of the LH-M test questions, a reasoning pattern absent from the training QAs, DiaKV keeps its gain: +3.1%, compared with +2.1% on the original questions (Table R3-2).

  • Larger models and other attention variants. DiaKV gains +4.1% to +10.1% on QASA-M with 24-69B models, including latent and native sparse attention (Table R2-7).

  • Augmentation without a stronger model. When the target model generates its own training QAs, DiaKV still improves 4 of 7 settings, and the rest stay close to the baseline (Table R2-3).

On the submitted results and claims

  • Format-free scoring on LH-M. Under LLM-as-judge, which ignores the output format, DiaKV remains on par with weight tuning: it is ahead in 8 of 9 comparisons (Table R3-1).
  • Positioning. We clarified our contribution in §1 and §2 of the revision. Training a KV cache is not new in itself; our contribution is to adapt a model to a general long context, beyond task-level pseudo-tokens (prefix tuning), compression (Cartridges, GradMem), and few-shot demonstrations (Context Tuning). See our responses on prefix tuning, Cartridges, and GradMem and on Context Tuning.

These are the questions on which your view would help us most:

  1. Do our preliminary experiments on extended prefix-cache reuse scenarios address the applicability concern?
  2. Do our additional experiments show that DiaKV's gain holds under other conditions: out-of-distribution queries, larger models with other attention variants, and augmentation without a stronger model?
  3. Is our claim appropriately scoped, both in relation to Cartridges and Context Tuning and in its position between test-time training and fine-tuning?

Even a one-line note on which points remain open would be valuable, and we will respond promptly to any request for clarification.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.