- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper proposes CacheWriter, a mechanism that trains a sender LLM to transmit a fixed number of key-value cache entries to a receiver LLM. The idea is an adaption of KV-Distill and involves training LoRA adapters on the sender’s key, value (and query and output) projections so that prepending the resulting cache entries to the receiver’s context produces an output distribution close to what the receiver would produce if it had access to the full private information. The authors compare their method with a number of selection-based baselines on various tasks. They show that adapting cache states help, particularly at settings where the transmission budget is tight. The paper then extensively tests whether improvements are the results of the sender's information, and under what conditions cachewritie is useful.
Critical strengths
1- The method is simple and easy to implement.
2- The existing experimental results are a major strengths of the paper and are quite insightful. This includes main results, controlled experiments, and ablations.
Critical weaknesses
1- The writing is strange and the paper is extremely hard to read. This seems to be the result of using an LLM to generate most of the discussions.
2- The paper provides a useful empirical study of cache communication, but the method is very similar to KV-Distill. In other words the contribution of the paper is the analysis rather than a new method.
3- There are two training-based baselines that are missing from this work. The paper should compare against a simple learned constant update (with the exact same loss and setting as CacheWriter) in order to justify the per-instance correction. The authors should also compare against KV-Distill. What I mean is a simple learned baseline is using KV-Distill to compress the sender’s private document into the same number of cache entries. Then pass them to the receiver. This would help show how better cacheWriter is compared to using an existing learned cache compressor to send the message.
Questions for authors
In addition to points raised in weaknesses:
1- How were the 32 or 64 positions chosen? What is the practical motivation of these values?
2- Can the authors provide results of scaling the number of positions budget to see whether CacheWriter remains competitive at more realistic budgets? It seems that the advantage of CahceWriter diminishes substantially at 64.
3- Did the authors observe any improvement from adapting all four projections over adapting only K and V? I'm trying to understand if there is benefit in the added training cost.
2 comments
Response to Reviewer 7k2a (2/2)
Q1. Why 32 and 64 entries. We chose these budgets for three reasons. First, they keep the message much shorter than the sender's private context, which is where the choice of message matters. 32 entries cover 29% of an average HotpotQA paragraph and 5% of five documents. Selection must then drop most of the context. This tight regime is a stricter test of whether a written message helps. Second, they match the budgets of our baselines. A 32-entry message fills as many receiver positions as a 32-token note. It also takes 28% of the full cache, close to the 30% that KVComm [1] sends. Third, our conclusions do not depend on this choice. CacheWriter leads selection at every budget from 32 to 256 entries with five documents, as the table under Q2 shows.
Q2. Larger budgets. To answer this question, we trained CacheWriter at 128 and 256 entries, in addition to 32 and 64. We use the five-document setting with Qwen3-8B on HotpotQA, where the private context is long enough for these budgets to matter. We then compare it with selection at each budget on test. CacheWriter leads selection at every budget. Its lead narrows as selection keeps more of the context, but it stays significant at 256 entries. CacheWriter also gets closer to the full context as the budget grows. The receiver scores 0.064 with no message and 0.278 when it reads all five of the sender's documents directly. CacheWriter closes 58% of this gap at 32 entries and 84% at 256.
| Five documents, Qwen3-8B on HotpotQA, test | 32 entries | 64 entries | 128 entries | 256 entries |
|---|---|---|---|---|
| Selection | 0.076 | 0.090 | 0.120 | 0.170 |
| CacheWriter | 0.189 | 0.196 | 0.215 | 0.244 |
| Share of the full-context gain recovered by CacheWriter | 58% | 62% | 71% | 84% |
To clarify, the smaller advantage at 64 entries that you mentioned appears only in the single-document setting, where the sender holds one paragraph. Our average lead over the strongest same-length baseline in this setting falls from +0.053 at 32 entries to +0.017 at 64. We still lead at 64 entries in 11 of the 16 cases, which cover four models on four benchmarks. The reason is that a paragraph is short, 112 to 191 tokens on average, depending on the benchmark. Selection at 64 entries already keeps the cache entries of a third to more than half of it, and with them much of the answer. Going from 32 to 64 entries raises the best selection policy by 0.089 and our message by 0.041. This is why we also test the five-document setting, where the sender holds five documents. They are much longer, and 64 entries cover less than 10% of them. Our message more than doubles the exact match of the best selection policy with five documents, from 0.094 to 0.196.
Q3. Key and value adapters only. We trained an adapter on the key and value projections alone, in the five-document setting, and compared it with our adapter on all four projections. The key-value adapter has 23.6M parameters against 61.3M for all four projections. Training time barely differs, 9.5 against 9.7 seconds per step on one NVIDIA L40S GPU (48 GB), because most of the compute goes to the frozen 8B base model. Inference time is the same. Writing a five-document message on the same GPU takes 0.073 s with either adapter merged into the sender's weights, against 0.072 s for running the sender without any adapter. The key-value adapter scores 0.162 on test at 32 entries, significantly below 0.189 for all four projections. The two match at 64 entries, 0.188 against 0.196. Adapting all four projections therefore helps most when the budget is tight and each entry must hold more of the context. We keep it as the default.
References
[1] KVComm: Enabling Efficient LLM Communication through Selective KV Sharing, Shi et al., ICLR 2026.
Response to Reviewer 7k2a (1/2)
Thank you for the constructive suggestions, which helped us make the comparison with existing methods more complete.
W1. Writing. We carefully revised the discussion of the results for readability. All changes aim to make our design and results easier to understand, while keeping the original experiments and main conclusions.
W2. Similarity to KV-Distill. The two methods differ in four respects. First, the reader differs. In KV-Distill [1], the model that compressed the context later reads it itself. In CacheWriter, a different agent reads the message. This receiver holds its own evidence and a question that the sender never sees. Second, the training target differs. KV-Distill's target is the model's own output on its full context. CacheWriter's target is the receiver's output when the receiver sees both its own evidence and the sender's full context. In our five-document setting, KV-Distill's objective leaves out the receiver's evidence and lowers test exact match from 0.196 to 0.143 at 64 entries. Third, the learned part differs. KV-Distill learns which tokens to keep, whereas CacheWriter fixes the kept positions and learns the content written there. A reimplementation of KV-Distill's learned selector reaches 0.162 at 32 entries, significantly below our 0.189, and it matches CacheWriter at 64 entries. Fourth, the scope differs. KV-Distill works within one model and does not cover communication across model families, whose caches differ in format. CacheWriter covers this case, and its message to a receiver of another family beats a same-length note by +0.120 and +0.179 on test. The two methods share a LoRA adapter and a mix of forward and reverse KL divergence. Both are common practice in fine-tuning and distillation and not specific to KV-Distill.
W3. The two trained baselines. We added both baselines to Table 2 of the revised PDF, using the five-document setting with Qwen3-8B on HotpotQA.
Learned constant update. Following your suggestion, we train one constant tensor that is added to every input's message, with exactly the same loss, data and setting as CacheWriter. We try two initialisations and two learning rates and keep the best on validation. This constant falls below CacheWriter by 0.051 at both 32 and 64 entries. Part of the update must therefore depend on the input.
KV-Distill as the message compressor. KV-Distill's code is not public. We therefore reimplement it in our setting, compress the sender's documents into the same number of cache entries, and pass them to the receiver. We test three variants, each a row of Table 2. The first trains the message with KV-Distill's objective, which ignores the receiver's evidence. The second trains it only to continue the sender's own text, without the receiver's evidence or question. The third uses KV-Distill's learned token selector. All three fall significantly below CacheWriter at 32 entries, scoring 0.116, 0.098 and 0.162 against 0.189. The first two also stay significantly below CacheWriter at 64 entries, at 0.143 and 0.122 against 0.196, while the learned selector comes closer at 0.179. Appendix E of the revised PDF gives the details.
References
[1] KV-Distill: Nearly Lossless Learnable Context Compression for LLMs, Chari et al., 2025.