- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper proposes DiaKV, which optimizes the cached context representation through supervised learning. Unlike traditional test-time training, which introduces additional latency by updating model parameters after a query arrives, DiaKV performs adaptation offline for contexts expected to be reused. Specifically, the method augments 20 seed question-answer pairs into 800 training examples and optimizes the corresponding KV cache through supervised learning.The results suggest that offline KV-cache optimization can improve context-specific QA performance without introducing additional online training latency.
Critical strengths
- The paper explores a promising direction by treating the KV cache as a trainable representation. This provides an interesting alternative to conventional weight-based fine-tuning and test-time training.
- The method achieves consistent improvements across multiple long-context QA benchmarks and language models. Although its practical applicability may be limited by the offline training overhead, the results demonstrate the potential of directly optimizing KV caches for context-specific adaptation.
Critical weaknesses
-
Although DiaKV improves long-context QA performance, it remains unclear whether these gains primarily come from acquiring task-specific knowledge, improving access to existing knowledge, or preserving the model's instruction-following capabilities. For example, on LongHealth-M, LoRA and full fine-tuning perform substantially worse than the untrained base model, partly due to response-format mismatches. This raises concerns about whether DiaKV's advantage over weight-based adaptation reflects better knowledge utilization or simply greater robustness to instruction-following degradation. Additional controlled experiments would help disentangle these factors.
-
DiaKV avoids online training latency by shifting adaptation to an offline stage, but still requires substantial computation to generate synthetic QAs and optimize the KV cache for each context. This limits its applicability to newly arriving or frequently updated documents, where repeated optimization may be costly. Although the method demonstrates the benefits of KV-cache optimization, the paper does not sufficiently analyze when the accuracy improvements justify the additional preparation cost, particularly under varying context reuse frequencies and update rates.
-
It remains unclear how well the optimized KV cache generalizes to queries that substantially differ from the training distribution. Although paper investigates the effects of seed QA coverage, it does not fully address generalization to unseen question types or reasoning patterns.
Questions for authors
-
Could the authors provide experiments that distinguish improvements in knowledge utilization from the preservation of instruction-following behavior, such as evaluating answer correctness independently of output-format compliance or using format-aligned training data?
-
Could the authors evaluate DiaKV on out-of-distribution queries, such as unseen question types or multi-hop reasoning tasks, to better understand its generalization beyond the augmented QA distribution?
4 comments
Author response (2/3)
[W2] Amortization analysis
Newly arriving and frequently updated context
"This limits its applicability to newly arriving or frequently updated documents, where repeated optimization may be costly."
Our motivation for DiaKV starts from a common characteristic of modern LLM serving systems — very high cache hit rate (often >90%; [1]) and increasing cache TTL (§2 - prefix caching & §4.3). We agree with your point that repeated optimization makes DiaKV costly to apply to newly arriving or frequently updated contexts. We want to clarify that such contexts incurring cache miss are not our primary targets. High cache hit rates of LLM serving systems suggest that DiaKV’s gain from cache hit contexts alone can still deliver a meaningful win for the entire system. Future work may study how to reuse DiaKV for a frequently updated context where the update is small and local. We believe such study deserves a separate work, and our work provides a groundwork for such investigation.
Amortization analysis
.. the paper does not sufficiently analyze when the accuracy improvements justify the additional preparation cost, particularly under varying context reuse frequencies and update rates.
The general formula for cost amortization is as follows, assuming the same accuracy with and without DiaKV:
- N = #(cache hit)
- is the discount factor for offline compute
- FC (Fixed Cost) = preparation cost for a single context (single DiaKV)
- MC (Marginal Cost) = per-query cost, serving with DiaKV
- C (Cost) = per-query cost, serving without DiaKV (i.e., a larger model matching DiaKV's accuracy)
The discount factor defines how much we discount for the offline resource. Focusing only on latency means = 0, as there is no “offline latency”. For a resource-constrained environment where offline resource use is as costly as online use, will be 1 or nearly 1.
It is hard to translate accuracy gain into cost measures like dollars or FLOPs. Our approach equates DiaKV’s accuracy gain from x to y with a cost gain: the cost of running a larger model whose accuracy is y minus the cost of running the target model whose accuracy is x. Note that the online cost of running target model with DiaKV is exactly identical to the online cost of running target model alone.
We use Llama-3.2-3B as an example target model. For preparation cost, we use USD 3.4 per context, which is the median over 4 datasets (Appendix C.3).
- FC = USD 3.4 (Appendix C.3)
- C - MC = USD 0.001 × (2.7 - 1) = USD 0.0017
- Llama-3.2-3B with DiaKV reaches 97.6% of the untrained Llama-3.1-8B's score in geometric mean over the four benchmarks.
- Assume running a single query on Llama-3.2-3B costs 0.1 cents and the cost gap between the two is proportional to the model size gap (2.7x larger)
Under this illustrative assumption, with α = 1, the preparation cost is amortized after approximately 2,000 cache hits. This is a very crude analysis, but we don’t believe the conclusion would easily change (it is an unfavorable conclusion for us, after all) — cost amortization requires heavy reuse in DiaKV.
As stated in the footnote on page 5, we have not claimed lower total cost in any part of the paper. Nevertheless, it would’ve been better to include a cost amortization analysis and honestly state that cost amortization requires heavy reuse in DiaKV.
DiaKV’s primary benefit is improved answer quality without additional online training latency, rather than lower total cost. This may be suitable for some (but not all) of the real-world scenarios where resources are relatively ample (so additional offline preparation cost is affordable) but maximizing accuracy and minimizing latency still matter. Prior works like sleep-time compute [2] or offline recurrence [3] also mostly leave out offline cost and prioritize online latency and accuracy. We will state our claim scope more clearly and report the amortization calculation only as an illustrative example.
[1] TraceLab: Characterizing Coding Agent Workloads for LLM Serving (Zhu et al., arXiv 2026)
[2] Sleep-time Compute: Beyond Inference Scaling at Test-time (Lin et al., arXiv 2025)
[3] Language Models Need Sleep (Lee et al., SPIGM Workshop @ ICML 2026)
Author response (3/3)
[W3 & Q2] Out-of-distribution, multi-hop queries
As the review recognizes, our §6.2 analysis addresses seed bias from locality, but does not fully address seed bias concern from other factors, including the factors the reviewer mentions — question types and reasoning patterns, e.g., multi-hop reasoning.
To address this weakness & question, we set up a controlled experiment on whether DiaKV’s gain extends to multi-hop queries, which is out-of-distribution to the training questions.
Our study is done on LH-M (LongHealth-M), one of the datasets we used in the paper (§4.1). A single LH-M context contains 10 patient cases, each including the patient name and his/her clinical documents. Most of the training questions (1957 of 1960) and the test questions (339 of 360) directly ask about a patient, referring to them by name. We consider such questions as single-hop questions. For these 339 single-hop test questions, we create multi-hop test questions by replacing the patient name with different facts about the patient. For example, if the original single-hop test question is:
When was the total spine MRI plain conducted in Laura Miller?
then the constructed multi-hop question is:
Consider the patient with hepatocellular carcinoma and liver cirrhosis. When was the total spine MRI plain conducted in this patient?
These questions are multi-hop as they require (1) identifying the patient among 10 given patients (who has hepatocellular carcinoma and liver cirrhosis? → Laura Miller) and then (2) locating the fact about that patient (When was the total spine MRI plain conducted in Laura Miller? → 2020-04-06). The answer remains identical to the single-hop question.
On the three models we’ve used in the paper (§4.1), we evaluate the same DiaKV used in the paper, which are trained on mostly single-hop questions. The results on original single-hop test questions and newly added multi-hop questions are in Table R3-2. Across three models, the mean accuracy gain in multi-hop questions is +3.1%, even higher than that of single-hop questions, +2.1%. The trends are similar for an auxiliary metric, word-level F1. This suggests that DiaKV’s gains transfer to reasoning patterns unseen in the training set, multi-hop reasoning in this case.
Table R3-2. LH-M results on the 339 multi-hop (out-of-distribution; OOD) vs single-hop questions (in-distribution; ID)
| Model | Questions | Acc. untrained | Acc. trained | F1 untrained | F1 trained |
|---|---|---|---|---|---|
| Qwen3-4B-2507 | single-hop (ID, original) | 0.708 | 0.729 (+3.0%) | 0.802 | 0.825 (+2.9%) |
| multi-hop (OOD) | 0.696 | 0.708 (+1.7%) | 0.786 | 0.813 (+3.4%) | |
| Llama-3.2-3B | single-hop | 0.478 | 0.487 (+1.9%) | 0.602 | 0.635 (+5.5%) |
| multi-hop | 0.445 | 0.490 (+10.1%) | 0.584 | 0.639 (+9.4%) | |
| Ministral-3-3B | single-hop | 0.667 | 0.678 (+1.6%) | 0.790 | 0.809 (+2.4%) |
| multi-hop | 0.687 | 0.690 (+0.4%) | 0.794 | 0.809 (+1.9%) | |
| Mean of 3 | single-hop | 0.618 | 0.631 (+2.1%) | 0.732 | 0.756 (+3.3%) |
| multi-hop | 0.610 | 0.629 (+3.1%) | 0.721 | 0.754 (+4.6%) |
Note that the accuracy and F1 for single-hop questions may slightly differ from those reported in the paper, as these results are on the 339-question subset (out of 360 in total) for comparison.
Thank you for your detailed response and additional experiments. These have addressed several of my concerns. I have therefore decided to raise my score to 3.
Author response (1/3)
We thank the reviewer for your valuable feedback. We group your reviews into three buckets based on their content.
[W1 & Q1] Disentangling the accuracy result of DiaKV vs weight tuning
Thank you for raising an important point in our evaluation. Even though the degradation of instruction-following capability in weight tuning (§5.2) is a valuable analysis, it may conflate the benefit DiaKV brings. Therefore, following your suggestion, we evaluate the LH-M (LongHealth-M) result on answer correctness alone, independent of output-format compliance. We have tried multiple rule-based metrics, but none of them fully isolates format compliance (examples: 07/13/2019 vs 07.13.19, Right ventricular anterior (RIVA) region vs RIVA area). We use LLM-as-judge instead, with the same setup as our two other datasets, QASA-M and FinTextQA (§4.1).
Results are in the table below. Losing instruction-following capability in weight tuning explains nearly all of the severe accuracy degradation in LH-M. However, even for LLM-as-judge, which does not penalize format at all, DiaKV remains on par with weight tuning. DiaKV outperforms all 3 weight baselines (full fine-tuning, LoRA r=32, LoRA r=8) in Qwen3 and Ministral, while only falling behind LoRA r=32 in Llama by some margin.
Table R3-1. Format-free (LLM-as-judge) evaluation on LH-M
Overall, format-free evaluation on LH-M shows that even factoring out the instruction-following failure, DiaKV remains competitive compared to weight tuning baselines. For clarity, we reiterate our claim in the paper regarding DiaKV vs weight tuning (§5.2): DiaKV is strictly better than weight tuning in terms of serving, while being “as effective as training weights” (L365). This format-free evaluation result does not alter our claim.