- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper suggest a compression method for KV cache in muti agent (with a different LLMs) setup called CacheCodec. In particular, the sharer's KV is compressed into learned latent space which is communicated to a (frozen) receiver. Then, the authors suggest a sequence DCT for frequency dependent quantization with straight-through gradient, and lossless entropy encoding (Zlip), so the compressed KV cache can be treated as a stream.
The method was evaluated on 3 sharers, one fixed (0.6 B) receiver on four multi-choice benchmarks. The paper show large saving vs. eaw cache transmission, while maintaining the accuracy (and even improved in some cases)/
Critical strengths
- The paper addresses relevant and timely question. How to send the model (latent) KV to a different model whose latent size is possibly different, under bandwidth constraint, while optimizing the task performance.
- Supporting prefix concatenation and C2C fusion is significant. The separation of learned representation compression from the bit steam coding has merit.
- The evaluation accross three sharers and four benchmark is useful starting point for evaluating heterogeneous communication networks. The paper reports the actual (serialized) payloads and decoding metadata (with encoding / decoding costs). The various bandwidth emphasize the tradeoff between the compression and overhead.
Critical weaknesses
-
The comparisons in the paper do not isolate the contribution of each part in the proposed coding pipeline. The main baseline is uncompressed KV, while the DCT, quantization and entropy coding are introduced together. A simple compressed baseline of LLC + quantization + zlib (without the DCT) would help to understand which components are necessary and what they contribute for task performance.
-
The communication protocol is not sufficiently described to assess the raw text (+compute) comparison. The paper names sharer context and receiver input but does not explain their content or overlap. The KV paths seemly communicate prefill KV states, whereas T2T (according to the table) pays for generating an intermediate message. This can be a meaningful comparison if the goal is to transfer the sharer's computed features without generating tokens, but it does not stress the advantage over forwarding existing context and prefilling it at the receiver. If the application requires the sender reasoning/action rollout, that computation is common work for both interfaces and should not be taken into account as a saving of KV communication.
-
Some clarification is needed about the deployment regime and latency accounting. The paper main result uses 50 Mbps link, while the small model pair communicate the raw KV at ~134-151 Mbps in Fig. 8. Moreover, Table 2demonstrates a benefit at 400 Mbps for the larger sharer. If so, then the result is pair dependent and not a general network speed claim. A concrete deployment and a cleaner timing protocol would help understanding the practical relevance. The aggregation behind the 8.64 speedup at the headline, while omitting the timing terms in Table 2 also need reconciliation.
4.The evidence for broader generality and small accuracy gains are quite limited. All pairs share one 0.6B receiver, the tasks are multiple choice, and the uncertainty is missing from the paper. The joint/split comparison also effect the parameters count, so its small gain does not isolate the effect of jointly encoding K and V.
Questions for authors
-
What exactly do and contain during training and evaluation? Does the receiver see the sharer's source context? Please specify the T2T prompt, actual message length, and the operation timed as you did at 1054 ms of 'sharer token re-encoding' part. Is this autoregressive message generation? If forwarding the existing plain text is information equivalent, how does that alternative compare? If receiver only already implements it, please write it explicitly.
-
Can you provide a comparison with simple quantized/compressed raw KV cache and with LLC + quantization + zlip without DCT, reporting both task accuracy and actual bytes volume that is sent? These would directly test the value of the proposed frequency-domain coding rather than compression in general.
-
Which deployment motivates the 50 Mbps operation point? please clarify whether the transmission is measured, emulated or estimated. Further, how the device transfer the synchronization are counted? How is the x8.64 headline aggregated? From Table 1 values, I obtain approximately 8.29x by averaging paired speedups, or 8.13x from summed latencies. In Table2, subtracting the listed components leaves 74 vs 64 ms for raw vs. codec concatenation, tough, the omitted terms are described as identical. What accounts for this difference ?
-
Are the encoder and decoder trained separately for each sharer-receiver pair? which modules are shared across layers? Please clarify the quantization scale beta and the evaluation split/sample counts. Uncertainty estimates or limited matched capacity control would help to establish whether small accuracy gains reflect reliable transfer.
6 comments
W3,Q3: Deployment regime and latency accounting.
W3: Some clarification is needed about the deployment regime and latency accounting. The paper main result uses 50 Mbps link, while the small model pair communicate the raw KV at ~134-151 Mbps in Fig. 8. Moreover, Table 2demonstrates a benefit at 400 Mbps for the larger sharer. If so, then the result is pair dependent and not a general network speed claim. A concrete deployment and a cleaner timing protocol would help understanding the practical relevance. The aggregation behind the 8.64 speedup at the headline, while omitting the timing terms in Table 2 also need reconciliation.
Q3: Which deployment motivates the 50 Mbps operation point? please clarify whether the transmission is measured, emulated or estimated. Further, how the device transfer the synchronization are counted? How is the x8.64 headline aggregated? From Table 1 values, I obtain approximately 8.29x by averaging paired speedups, or 8.13x from summed latencies. In Table2, subtracting the listed components leaves 74 vs 64 ms for raw vs. codec concatenation, tough, the omitted terms are described as identical. What accounts for this difference?
We clarify that 50 Mbps is a controlled bandwidth-constrained operating point motivated by communication between distributed LLM workers on edge or edge–cloud nodes. The latency benefit depends on the model pair: both the transmitted KV volume and the additional processing cost determine the crossover bandwidth. The approximately 134–151 Mbps crossover points in Fig. 8 apply specifically to the smaller model pair. Table 2 examines Qwen3-4B-Base → Qwen3-0.6B at 400 Mbps, where the larger raw KV payload provides sufficient transmission savings to offset the additional processing.
Our timing protocol combines measured local operations with a bandwidth-dependent link model. Serialization, socketpair communication, framing, and deserialization operate on the actual payloads and are timed. The link delay is determined by , where is the serialized size in bytes and is the configured rate in bits/s. E2E latency covers the pipeline from immediately before Sharer prefill to completion of the Receiver’s answer generation or selection and evaluator processing. Synchronization waits required to complete GPU operations and device transfers are included within the corresponding measured intervals; timers are stopped only after those operations have completed. Required device-to-host copies are included in sender-side encoding, while host-to-device reconstruction is included in receiver-side decoding. The uncompressed C2C and CacheCodec paths use the same E2E boundaries and accounting rules.
Our reported 8.64× is the arithmetic mean of paired speedups across 12 Fusion configurations (three model pairs × four benchmarks).
where indexes model pairs, indexes benchmarks, and and denote the E2E latencies of uncompressed C2C and CacheCodec under Fusion. For these same configurations, the ratio of summed latencies is 8.22×. We agree with your calculations of 8.29× and 8.13×, respectively, which aggregate both Concat and Fusion across all 24 configurations. Both sets of results are valid under their respective aggregation scopes. We will clarify the Fusion-only scope of the headline and specify the aggregation formula in the manuscript.
Regarding the omitted terms in Table 2, we clarify that the two pipelines contain common categories of operations, but their measured durations differ. The breakdown below accounts for 73.334 ms in Raw Concat and 64.315 ms in CacheCodec Concat. The main differences are a 16.038 ms reduction in non-link transport overhead and a 7.570 ms increase in additional sender processing. For Raw Concat, the 16.445 ms transport overhead comprises serialization (6.762 ms), socket runtime (4.016 ms), and deserialization (5.667 ms), separately from the 324.509 ms modeled link delay. These measured differences explain why the remaining latency is unequal.
| Previously omitted component | Raw Concat (ms) | CacheCodec Concat (ms) |
|---|---|---|
| Sharer prefill | 27.892 | 27.922 |
| Additional sender processing | 2.637 | 10.207 |
| Transport overhead | 16.445 | 0.407 |
| Additional receiver reconstruction | 10.491 | 10.155 |
| Receiver prefill | 15.033 | 14.796 |
| Token selection & evaluator overhead | 0.836 | 0.828 |
| Total omitted latency | 73.334 | 64.315 |
W4,Q4: Broader evaluation and implementation details.
W4: The evidence for broader generality and small accuracy gains are quite limited. All pairs share one 0.6B receiver, the tasks are multiple choice, and the uncertainty is missing from the paper. The joint/split comparison also effect the parameters count, so its small gain does not isolate the effect of jointly encoding K and V.
Q4: Are the encoder and decoder trained separately for each sharer-receiver pair? which modules are shared across layers? Please clarify the quantization scale beta and the evaluation split/sample counts. Uncertainty estimates or limited matched capacity control would help to establish whether small accuracy gains reflect reliable transfer.
We extended the evaluation to an additional Receiver and LongBench-E[1] tasks. For the reversed Qwen pair (Qwen3-0.6B → Qwen2.5-0.5B-Instruct), CacheCodec improves four-benchmark average accuracy from 39.38% to 40.65% and reduces average E2E latency from 538.8 to 174.0 ms compared with uncompressed C2C, yielding a 3.10× speedup based on average latencies.
Entries report accuracy (%) / E2E latency (ms).
| Method | MMLU-Redux | OpenBookQA | ARC-C | C-Eval | Average |
|---|---|---|---|---|---|
| C2C | 36.5 / 595 | 42.2 / 459 | 40.7 / 534 | 38.1 / 567 | 39.38 / 538.8 |
| CacheCodec | 38.1 / 171 | 42.8 / 173 | 41.9 / 193 | 39.8 / 159 | 40.65 / 174.0 |
On the LongBench-E[1] tasks, CacheCodec improves the task score from 38.99 to 45.26. These tasks cover single-document QA, multi-document QA, few-shot learning and synthetic tasks, extending the evidence beyond multiple-choice evaluation. Together, these results support the applicability of CacheCodec across additional Receiver and task settings.
Values are reported as Uncompressed C2C → CacheCodec.
| Task Type | Avg. Length | Task Score ↑ | E2E (s) ↓ | Payload (KB) ↓ | Speedup ↑ |
|---|---|---|---|---|---|
| Single-document QA | 4.09K | 29.96 → 37.59 | 23.73 → 12.77 | 77,168 → 126.8 | 1.86× |
| Multi-document QA | 7.02K | 33.23 → 39.11 | 29.90 → 22.43 | 97,888 → 152.6 | 1.33× |
| Few-shot Learning | 6.55K | 48.75 → 55.11 | 26.01 → 16.61 | 94,372 → 147.8 | 1.57× |
| Synthetic Tasks | 9.24K | 39.16 → 44.31 | 22.89 → 19.13 | 93,977 → 148.8 | 1.20× |
| All Tasks | 6.70K | 38.99 → 45.26 | 26.20 → 18.32 | 93,058 → 146.6 | 1.43× |
Datasets: Single-document QA: Qasper, MultiFieldQA-en; Multi-document QA: HotpotQA, 2WikiMQA; Few-shot Learning: TREC, TriviaQA, SAMSum; Synthetic Tasks: PassageCount, PassageRetrieval-en;
The Joint/Split ablation controls the communication budget rather than the parameter count: Joint uses a shared -dimensional latent, while Split allocates dimensions to K and V separately, keeping the total transmitted representation size identical. Under this matched budget, Joint improves accuracy on all four benchmarks (45.8% → 46.4% on average). Moreover, our CCA analysis in Appendix A.3 shows substantially stronger canonical correlations for aligned K/V pairs than for shuffled pairs, revealing non-trivial shared structure between K and V. Together, these results support the shared latent design: Joint can model this cross-K/V structure within the same communication budget, although its architecture naturally introduces additional parameters. This comparison does not isolate the effect of joint encoding from the difference in parameter count.
For each Sharer–Receiver pair and integration mechanism, the LCC encoder and Receiver-side K/V reconstruction mappings are trained jointly end-to-end using the Receiver’s task loss, while both LLMs remain frozen. The learned communication parameters are specific to that configuration. Each mapped layer pair has its own LCC and reconstruction parameters, without weight sharing across layers. The Frequency Codec uses the same fixed DCT operation, quantization schedule, and entropy-coding configuration across layers. Both C2C and CacheCodec are trained and evaluated under the same experimental settings.
As shown in Eq. (5), globally scales the quantization step sizes, with larger values producing coarser quantization. The fixed frequency schedule assigns finer steps to lower frequencies and is shared across layers and between K and V. The forward pass uses hard rounding, with a straight-through estimator (STE) during training. We evaluate the test splits of OpenBookQA (500/500; no filtering), ARC-C (1,150/1,172; excluding gold answers not mappable to A/B/C/D), MMLU-Redux (5,632/5,700; excluding no_correct_answer and expert, and using correct_answer for wrong_groundtruth), and C-Eval (12,342/12,342; no filtering). The four-benchmark average weights each benchmark equally.
[1] Bai et al., “LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding,” ACL 2024.
We sincerely thank Reviewer DTdJ for the detailed technical feedback.
W1,Q2: Component-wise Frequency Codec ablation and direct compression controls
Compared with K2V2 and K4V4, CacheCodec improves accuracy by 1.90 and 1.95 percentage points (pp) while reducing payload by 109.4× and 218.2×, respectively. Removing DCT decreases accuracy from 47.90% to 44.53% and increases payload from 11,438.5 to 15,417.1 bytes (+34.8%). This improvement is consistent with DCT concentrating signal energy into fewer dominant frequency coefficients, enabling more effective quantization and entropy coding. Furthermore, unlike conventional cache compressors, CacheCodec learns compact representations through the frozen Receiver's downstream task loss, without requiring explicit cache reconstruction.
W2, Q1: Communication protocol and T2T latency.
For the KV paths, Xs and Xr are constructed from the same OpenHermes conversation during training and contain the same question, choices, and answer-format instruction during evaluation, using model-specific chat templates and tokenizers. The reported Receiver-only baseline covers the information and Receiver computation required by the source-text alternative; no additional text transfer is needed because that content is already available locally.
Across all three Sharer models, CacheCodec achieves higher average accuracy than both the Sharer-only and Receiver-only baselines. These results support the utility of sharing computed representations even with identical source content: heterogeneous models can extract different features from the same input, and CacheCodec’s Receiver-supervised training enables the transmitted representation to support the Receiver’s downstream predictions.
For KV communication, the Sharer performs only a prompt prefill, without generating intermediate text or a reasoning rollout. T2T prompts the Sharer with “In one clear sentence, describe the most essential background knowledge needed to answer the question: {question}” and autoregressively generates a bridge message averaging 28.39 tokens. The transmitted payload is the generated UTF-8 text. The Receiver then processes the background request, the generated message, and the complete multiple-choice prompt.
The 1054 ms entry in Table 2 measures Sharer autoregressive message generation; we will replace “sharer token re-encoding” with “Sharer message generation.” This message is generated specifically for T2T communication. If an application independently requires a reasoning or action rollout, its computation must be accounted for consistently across interfaces and cannot be attributed to KV communication savings. Appendix A.1 will provide the complete training and evaluation prompt constructions with a concrete example.