CacheWriter: Learning to Write Latent Messages for Frozen Language Models

Large language model (LLM) agents often hold private information, so a shared task requires one agent to pass what it knows to another. Text is the usual channel, but it costs the sender a decode and the receiver a re-encode, and the message must fit into the receiver's context. Transmitting key–value (KV) cache entries avoids the decode. However, existing cache handoff only selects among states that the sender computed for its own continuation, and under a tight position budget, selection leaves most of the available gain unrecovered. In this work, we propose CacheWriter, in which a question-blind sender writes a fixed number of KV entries for a frozen receiver. Specifically, each entry is a selected state plus a zero-initialised low-rank update, so that training starts exactly at selection. The update learns to match the receiver's output distribution given the full private context, without any answer labels. Extensive experiments across four model families and four single-document question-answering benchmarks demonstrate that 32-entry messages improve validation exact match by 5.3 percentage points on average over the strongest position-matched baseline. On answer-exclusive, randomly ordered five-document HotpotQA, 64 written entries more than double the best selection policy for Qwen3-8B (19.6% vs. 9.4% held-out exact match). Replacing the learned update with its document-independent mean further shows that the instance-specific part of the gain grows as the budget binds. Across model families, one fixed calibration per pair beats a position-matched text note by 8.6 points (Llama-3.1→Qwen3) and 14.6 points (Qwen3→Falcon3) of held-out exact match, and on the first pair it is statistically indistinguishable from same-family writing.