acceptodds
Under review as a conference paper at ICLR 2027

CacheCodec: Task-Oriented KV Compression for Bandwidth-Constrained Multi-LLM Communication

Abstract

Large language models are increasingly deployed in collaborative systems, where exchanging key-value (KV) states can bypass autoregressive intermediate-text generation and preserve rich model-internal context. However, KV caches are high-dimensional, model-specific states designed for local inference rather than communication, making direct transmission prohibitively expensive over bandwidth-constrained links. We therefore formulate KV compression as task-oriented communication, preserving downstream-relevant information rather than faithfully reconstructing the transmitted KV states. Based on this perspective, we propose CacheCodec, a task-oriented codec for efficient inter-LLM KV communication. CacheCodec first maps high-dimensional KV states into a compact representation message optimized under downstream task supervision, and then applies frequency-domain coding along the sequence dimension to convert this representation into a low-rate bitstream for transmission. Across heterogeneous LLM pairs and downstream tasks, CacheCodec reduces transmitted payload by while maintaining or improving task accuracy, and achieves reduction in end-to-end latency over a 50 Mbps communication link.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multi-Agent Systems, Multi-LLM Communication, KV Compression

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_DTdJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper suggest a compression method for KV cache in muti agent (with a different LLMs) setup called CacheCodec. In particular, the sharer's KV is compressed into learned latent space which is communicated to a (frozen) receiver. Then, the authors suggest a sequence DCT for frequency dependent quantization with straight-through gradient, and lossless entropy encoding (Zlip), so the compressed KV cache can be treated as a stream.

The method was evaluated on 3 sharers, one fixed (0.6 B) receiver on four multi-choice benchmarks. The paper show large saving vs. eaw cache transmission, while maintaining the accuracy (and even improved in some cases)/

Critical strengths
  • The paper addresses relevant and timely question. How to send the model (latent) KV to a different model whose latent size is possibly different, under bandwidth constraint, while optimizing the task performance.
  • Supporting prefix concatenation and C2C fusion is significant. The separation of learned representation compression from the bit steam coding has merit.
  • The evaluation accross three sharers and four benchmark is useful starting point for evaluating heterogeneous communication networks. The paper reports the actual (serialized) payloads and decoding metadata (with encoding / decoding costs). The various bandwidth emphasize the tradeoff between the compression and overhead.
Critical weaknesses
  1. The comparisons in the paper do not isolate the contribution of each part in the proposed coding pipeline. The main baseline is uncompressed KV, while the DCT, quantization and entropy coding are introduced together. A simple compressed baseline of LLC + quantization + zlib (without the DCT) would help to understand which components are necessary and what they contribute for task performance.

  2. The communication protocol is not sufficiently described to assess the raw text (+compute) comparison. The paper names sharer context xsx_s and receiver input xrx_r but does not explain their content or overlap. The KV paths seemly communicate prefill KV states, whereas T2T (according to the table) pays for generating an intermediate message. This can be a meaningful comparison if the goal is to transfer the sharer's computed features without generating tokens, but it does not stress the advantage over forwarding existing context and prefilling it at the receiver. If the application requires the sender reasoning/action rollout, that computation is common work for both interfaces and should not be taken into account as a saving of KV communication.

  3. Some clarification is needed about the deployment regime and latency accounting. The paper main result uses 50 Mbps link, while the small model pair communicate the raw KV at ~134-151 Mbps in Fig. 8. Moreover, Table 2demonstrates a benefit at 400 Mbps for the larger sharer. If so, then the result is pair dependent and not a general network speed claim. A concrete deployment and a cleaner timing protocol would help understanding the practical relevance. The aggregation behind the 8.64 speedup at the headline, while omitting the timing terms in Table 2 also need reconciliation.

4.The evidence for broader generality and small accuracy gains are quite limited. All pairs share one 0.6B receiver, the tasks are multiple choice, and the uncertainty is missing from the paper. The joint/split comparison also effect the parameters count, so its small gain does not isolate the effect of jointly encoding K and V.

Questions for authors
  1. What exactly do xsx_s and xrx_r contain during training and evaluation? Does the receiver see the sharer's source context? Please specify the T2T prompt, actual message length, and the operation timed as you did at 1054 ms of 'sharer token re-encoding' part. Is this autoregressive message generation? If forwarding the existing plain text is information equivalent, how does that alternative compare? If receiver only already implements it, please write it explicitly.

  2. Can you provide a comparison with simple quantized/compressed raw KV cache and with LLC + quantization + zlip without DCT, reporting both task accuracy and actual bytes volume that is sent? These would directly test the value of the proposed frequency-domain coding rather than compression in general.

  3. Which deployment motivates the 50 Mbps operation point? please clarify whether the transmission is measured, emulated or estimated. Further, how the device transfer the synchronization are counted? How is the x8.64 headline aggregated? From Table 1 values, I obtain approximately 8.29x by averaging paired speedups, or 8.13x from summed latencies. In Table2, subtracting the listed components leaves 74 vs 64 ms for raw vs. codec concatenation, tough, the omitted terms are described as identical. What accounts for this difference ?

  4. Are the encoder and decoder trained separately for each sharer-receiver pair? which modules are shared across layers? Please clarify the quantization scale beta and the evaluation split/sample counts. Uncertainty estimates or limited matched capacity control would help to establish whether small accuracy gains reflect reliable transfer.

6 comments

We sincerely thank Reviewer DTdJ for the detailed technical feedback.

W1,Q2: Component-wise Frequency Codec ablation and direct compression controls

W1: The comparisons in the paper do not isolate the contribution of each part in the proposed coding pipeline. The main baseline is uncompressed KV, while the DCT, quantization and entropy coding are introduced together. A simple compressed baseline of LLC + quantization + zlib (without the DCT) would help to understand which components are necessary and what they contribute for task performance.

Q2: Can you provide a comparison with simple quantized/compressed raw KV cache and with LLC + quantization + zlip without DCT, reporting both task accuracy and actual bytes volume that is sent? These would directly test the value of the proposed frequency-domain coding rather than compression in general.

Compared with K2V2 and K4V4, CacheCodec improves accuracy by 1.90 and 1.95 percentage points (pp) while reducing payload by 109.4× and 218.2×, respectively. Removing DCT decreases accuracy from 47.90% to 44.53% and increases payload from 11,438.5 to 15,417.1 bytes (+34.8%). This improvement is consistent with DCT concentrating signal energy into fewer dominant frequency coefficients, enabling more effective quantization and entropy coding. Furthermore, unlike conventional cache compressors, CacheCodec learns compact representations through the frozen Receiver's downstream task loss, without requiring explicit cache reconstruction.

MethodAccuracy (%) ↑Actual transmitted bytes/sample ↓
Uncompressed raw KV45.909,985,626
K4V445.952,496,406
K2V246.001,251,331
CacheCodec w/o DCT44.5315,417.1
CacheCodec47.9011,438.5

W2, Q1: Communication protocol and T2T latency.

W2: The communication protocol is not sufficiently described to assess the raw text (+compute) comparison. The paper names sharer context Xs and receiver input Xr but does not explain their content or overlap. The KV paths seemly communicate prefill KV states, whereas T2T (according to the table) pays for generating an intermediate message. This can be a meaningful comparison if the goal is to transfer the sharer's computed features without generating tokens, but it does not stress the advantage over forwarding existing context and prefilling it at the receiver. If the application requires the sender reasoning/action rollout, that computation is common work for both interfaces and should not be taken into account as a saving of KV communication.

Q1: What exactly do Xs and Xr contain during training and evaluation? Does the receiver see the sharer's source context? Please specify the T2T prompt, actual message length, and the operation timed as you did at 1054 ms of 'sharer token re-encoding' part. Is this autoregressive message generation? If forwarding the existing plain text is information equivalent, how does that alternative compare? If receiver only already implements it, please write it explicitly.

For the KV paths, XsX_s and XrX_r are constructed from the same OpenHermes conversation during training and contain the same question, choices, and answer-format instruction during evaluation, using model-specific chat templates and tokenizers. The reported Receiver-only baseline covers the information and Receiver computation required by the source-text alternative; no additional text transfer is needed because that content is already available locally.

Across all three Sharer models, CacheCodec achieves higher average accuracy than both the Sharer-only and Receiver-only baselines. These results support the utility of sharing computed representations even with identical source content: heterogeneous models can extract different features from the same input, and CacheCodec’s Receiver-supervised training enables the transmitted representation to support the Receiver’s downstream predictions.

For KV communication, the Sharer performs only a prompt prefill, without generating intermediate text or a reasoning rollout. T2T prompts the Sharer with “In one clear sentence, describe the most essential background knowledge needed to answer the question: {question}” and autoregressively generates a bridge message averaging 28.39 tokens. The transmitted payload is the generated UTF-8 text. The Receiver then processes the background request, the generated message, and the complete multiple-choice prompt.

The 1054 ms entry in Table 2 measures Sharer autoregressive message generation; we will replace “sharer token re-encoding” with “Sharer message generation.” This message is generated specifically for T2T communication. If an application independently requires a reasoning or action rollout, its computation must be accounted for consistently across interfaces and cannot be attributed to KV communication savings. Appendix A.1 will provide the complete training and evaluation prompt constructions with a concrete example.

W3,Q3: Deployment regime and latency accounting.

W3: Some clarification is needed about the deployment regime and latency accounting. The paper main result uses 50 Mbps link, while the small model pair communicate the raw KV at ~134-151 Mbps in Fig. 8. Moreover, Table 2demonstrates a benefit at 400 Mbps for the larger sharer. If so, then the result is pair dependent and not a general network speed claim. A concrete deployment and a cleaner timing protocol would help understanding the practical relevance. The aggregation behind the 8.64 speedup at the headline, while omitting the timing terms in Table 2 also need reconciliation.

Q3: Which deployment motivates the 50 Mbps operation point? please clarify whether the transmission is measured, emulated or estimated. Further, how the device transfer the synchronization are counted? How is the x8.64 headline aggregated? From Table 1 values, I obtain approximately 8.29x by averaging paired speedups, or 8.13x from summed latencies. In Table2, subtracting the listed components leaves 74 vs 64 ms for raw vs. codec concatenation, tough, the omitted terms are described as identical. What accounts for this difference?

We clarify that 50 Mbps is a controlled bandwidth-constrained operating point motivated by communication between distributed LLM workers on edge or edge–cloud nodes. The latency benefit depends on the model pair: both the transmitted KV volume and the additional processing cost determine the crossover bandwidth. The approximately 134–151 Mbps crossover points in Fig. 8 apply specifically to the smaller model pair. Table 2 examines Qwen3-4B-Base → Qwen3-0.6B at 400 Mbps, where the larger raw KV payload provides sufficient transmission savings to offset the additional processing.

Our timing protocol combines measured local operations with a bandwidth-dependent link model. Serialization, socketpair communication, framing, and deserialization operate on the actual payloads and are timed. The link delay is determined by Tlink=8Bpayload/RlinkT_{\mathrm{link}}=8B_{\mathrm{payload}}/R_{\mathrm{link}}, where BpayloadB_{\mathrm{payload}} is the serialized size in bytes and RlinkR_{\mathrm{link}} is the configured rate in bits/s. E2E latency covers the pipeline from immediately before Sharer prefill to completion of the Receiver’s answer generation or selection and evaluator processing. Synchronization waits required to complete GPU operations and device transfers are included within the corresponding measured intervals; timers are stopped only after those operations have completed. Required device-to-host copies are included in sender-side encoding, while host-to-device reconstruction is included in receiver-side decoding. The uncompressed C2C and CacheCodec paths use the same E2E boundaries and accounting rules.

Our reported 8.64× is the arithmetic mean of paired speedups across 12 Fusion configurations (three model pairs × four benchmarks).

S=112∑p=13∑d=14TC2C(p,d)TOurs(p,d)S=\frac{1}{12}\sum_{p=1}^{3}\sum_{d=1}^{4}\frac{T_{C2C}^{(p,d)}}{T_{Ours}^{(p,d)}}

where pp indexes model pairs, dd indexes benchmarks, and TC2CT_{C2C} and TOursT_{Ours} denote the E2E latencies of uncompressed C2C and CacheCodec under Fusion. For these same configurations, the ratio of summed latencies is 8.22×. We agree with your calculations of 8.29× and 8.13×, respectively, which aggregate both Concat and Fusion across all 24 configurations. Both sets of results are valid under their respective aggregation scopes. We will clarify the Fusion-only scope of the headline and specify the aggregation formula in the manuscript.

Regarding the omitted terms in Table 2, we clarify that the two pipelines contain common categories of operations, but their measured durations differ. The breakdown below accounts for 73.334 ms in Raw Concat and 64.315 ms in CacheCodec Concat. The main differences are a 16.038 ms reduction in non-link transport overhead and a 7.570 ms increase in additional sender processing. For Raw Concat, the 16.445 ms transport overhead comprises serialization (6.762 ms), socket runtime (4.016 ms), and deserialization (5.667 ms), separately from the 324.509 ms modeled link delay. These measured differences explain why the remaining latency is unequal.

Previously omitted componentRaw Concat (ms)CacheCodec Concat (ms)
Sharer prefill27.89227.922
Additional sender processing2.63710.207
Transport overhead16.4450.407
Additional receiver reconstruction10.49110.155
Receiver prefill15.03314.796
Token selection & evaluator overhead0.8360.828
Total omitted latency73.33464.315

W4,Q4: Broader evaluation and implementation details.

W4: The evidence for broader generality and small accuracy gains are quite limited. All pairs share one 0.6B receiver, the tasks are multiple choice, and the uncertainty is missing from the paper. The joint/split comparison also effect the parameters count, so its small gain does not isolate the effect of jointly encoding K and V.

Q4: Are the encoder and decoder trained separately for each sharer-receiver pair? which modules are shared across layers? Please clarify the quantization scale beta and the evaluation split/sample counts. Uncertainty estimates or limited matched capacity control would help to establish whether small accuracy gains reflect reliable transfer.

We extended the evaluation to an additional Receiver and LongBench-E[1] tasks. For the reversed Qwen pair (Qwen3-0.6B → Qwen2.5-0.5B-Instruct), CacheCodec improves four-benchmark average accuracy from 39.38% to 40.65% and reduces average E2E latency from 538.8 to 174.0 ms compared with uncompressed C2C, yielding a 3.10× speedup based on average latencies.

Entries report accuracy (%) / E2E latency (ms).

MethodMMLU-ReduxOpenBookQAARC-CC-EvalAverage
C2C36.5 / 59542.2 / 45940.7 / 53438.1 / 56739.38 / 538.8
CacheCodec38.1 / 17142.8 / 17341.9 / 19339.8 / 15940.65 / 174.0

On the LongBench-E[1] tasks, CacheCodec improves the task score from 38.99 to 45.26. These tasks cover single-document QA, multi-document QA, few-shot learning and synthetic tasks, extending the evidence beyond multiple-choice evaluation. Together, these results support the applicability of CacheCodec across additional Receiver and task settings.

Values are reported as Uncompressed C2C → CacheCodec.

Task TypeAvg. LengthTask Score ↑E2E (s) ↓Payload (KB) ↓Speedup ↑
Single-document QA4.09K29.96 → 37.5923.73 → 12.7777,168 → 126.81.86×
Multi-document QA7.02K33.23 → 39.1129.90 → 22.4397,888 → 152.61.33×
Few-shot Learning6.55K48.75 → 55.1126.01 → 16.6194,372 → 147.81.57×
Synthetic Tasks9.24K39.16 → 44.3122.89 → 19.1393,977 → 148.81.20×
All Tasks6.70K38.99 → 45.2626.20 → 18.3293,058 → 146.61.43×

Datasets: Single-document QA: Qasper, MultiFieldQA-en; Multi-document QA: HotpotQA, 2WikiMQA; Few-shot Learning: TREC, TriviaQA, SAMSum; Synthetic Tasks: PassageCount, PassageRetrieval-en;

The Joint/Split ablation controls the communication budget rather than the parameter count: Joint uses a shared dzd_z-dimensional latent, while Split allocates dz/2d_z/2 dimensions to K and V separately, keeping the total transmitted representation size identical. Under this matched budget, Joint improves accuracy on all four benchmarks (45.8% → 46.4% on average). Moreover, our CCA analysis in Appendix A.3 shows substantially stronger canonical correlations for aligned K/V pairs than for shuffled pairs, revealing non-trivial shared structure between K and V. Together, these results support the shared latent design: Joint can model this cross-K/V structure within the same communication budget, although its architecture naturally introduces additional parameters. This comparison does not isolate the effect of joint encoding from the difference in parameter count.

For each Sharer–Receiver pair and integration mechanism, the LCC encoder and Receiver-side K/V reconstruction mappings are trained jointly end-to-end using the Receiver’s task loss, while both LLMs remain frozen. The learned communication parameters are specific to that configuration. Each mapped layer pair has its own LCC and reconstruction parameters, without weight sharing across layers. The Frequency Codec uses the same fixed DCT operation, quantization schedule, and entropy-coding configuration across layers. Both C2C and CacheCodec are trained and evaluated under the same experimental settings.

As shown in Eq. (5), β\beta globally scales the quantization step sizes, with larger values producing coarser quantization. The fixed frequency schedule assigns finer steps to lower frequencies and is shared across layers and between K and V. The forward pass uses hard rounding, with a straight-through estimator (STE) during training. We evaluate the test splits of OpenBookQA (500/500; no filtering), ARC-C (1,150/1,172; excluding gold answers not mappable to A/B/C/D), MMLU-Redux (5,632/5,700; excluding no_correct_answer and expert, and using correct_answer for wrong_groundtruth), and C-Eval (12,342/12,342; no filtering). The four-benchmark average weights each benchmark equally.

[1] Bai et al., “LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding,” ACL 2024.

Reviewer_wK6tOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes CacheCodec, a task-oriented codec for KV cache communication between heterogeneous LLMs. It compresses Sharer KV states into a compact latent with a receiver-supervised compressor and encodes the latent into a bitstream with DCT-based quantization and entropy coding.

Critical strengths

The paper formulates inter-LLM KV compression as a task-oriented communication problem, where the goal is to preserve the information the Receiver needs rather than to reconstruct the Sharer's KV states.

The proposed CacheCodec separates what to send from how to send it: a Latent Channel Compressor trained with the Receiver's task loss produces a compact representation, and a frequency codec turns it into a low-rate bitstream.

The evaluation reports actual serialized payload and end-to-end latency across heterogeneous Sharer–Receiver pairs and two cache integration methods, not just theoretical compression ratios.

Critical weaknesses

The paper does not compare against any existing KV compression method. The only baseline is uncompressed KV transmission.

All four benchmarks are short-context multiple-choice tasks with at most 16 generated tokens. Cross-model KV communication matters most for long-context and complex reasoning tasks, which are not evaluated.

The writing needs polishing. "Concatenation" is repeatedly misspelled as "Concatentation", the DeepSeek-V2 citation appears as "dee, 2024", the main text refers to "Figure A.5" instead of Figure 8, and Appendices A.2.2 and A.3 contain large blank spaces.

Questions for authors

See the above weaknesses.

3 comments

We sincerely thank Reviewer wK6t for the constructive feedback. We provide additional experiments and detailed revisions addressing each concern below.

W1: Comparison of Existing Compression Baselines

The paper does not compare against any existing KV compression method. The only baseline is uncompressed KV transmission.

Our additional comparisons with KIVI, CacheGen, and KVQuant show that CacheCodec achieves the highest average accuracy and lowest average end-to-end latency across all three evaluated model pairs. We integrate three representative KV-cache compression methods: KIVI, CacheGen, and KVQuant into the same cross-model communication pipeline, covering five compression configurations. Each method compresses the same Sharer KV cache, transmits the complete serialized payload including all method-specific metadata, reconstructs the cache. We use identical evaluation samples, decoding settings, bandwidth, and latency accounting. The reported E2E latency includes all codec and transport overheads.

Each entry below reports average accuracy (%) / average E2E latency (ms) over MMLU-Redux, OpenBookQA, ARC-C, and C-Eval.

Receiver for all three Sharers: Qwen3-0.6B.

MethodQwen2.5-0.5BQwen3-4BLlama-3.2-1B
Uncompressed C2C46.88 / 448.044.40 / 3751.045.90 / 1140.8
KIVI (K2V2)46.95 / 275.044.23 / 1295.046.00 / 404.0
KIVI (K4V4)46.78 / 306.544.38 / 1655.545.95 / 477.0
CacheGen46.70 / 324.543.90 / 1467.845.88 / 470.3
KVQuant (2-bit)46.53 / 272.344.83 / 909.045.73 / 330.3
KVQuant (4-bit)46.75 / 306.544.10 / 1343.045.88 / 420.0
CacheCodec48.18 / 267.546.40 / 217.347.90 / 164.8

The existing compressors achieve 1.38–4.13× E2E speedups over uncompressed C2C, with average accuracy changes ranging from −0.50 to +0.43 pp. CacheCodec exceeds the highest average accuracy among these compressed baselines by 1.23, 1.57, and 1.90 pp for the Qwen2.5-0.5B, Qwen3-4B, and Llama-3.2-1B Sharers, respectively, while providing an additional 1.02×, 4.18×, and 2.00× E2E speedup over the fastest compressed baseline for each pair.

These results demonstrate that CacheCodec simultaneously improves downstream task accuracy and reduces E2E latency compared with existing KV compressors in the evaluated heterogeneous C2C settings.

W2: Evaluation on Long-Context and Diverse Tasks

All four benchmarks are short-context multiple-choice tasks with at most 16 generated tokens. Cross-model KV communication matters most for long-context and complex reasoning tasks, which are not evaluated.

In our additional LongBench-E[1] evaluation, CacheCodec improves task scores and reduces E2E latency across all four evaluated task categories, with category-level average context lengths ranging from 4.09K to 9.24K tokens.

We extend our evaluation to LongBench-E[1] samples, covering complex reasoning tasks beyond multiple-choice benchmarks, including single-/multi-document QA, few-shot learning, and synthetic tasks. The results show that CacheCodec improves the overall task score from 38.99 to 45.26(+6.27 pp) while reducing average E2E latency from 26.20 s to 18.32 s (1.43× speedup).

Values are reported as Uncompressed C2C → CacheCodec.

Task TypeAvg. LengthTask Score ↑E2E (s) ↓Payload (KB) ↓Speedup ↑
Single-document QA4.09K29.96 → 37.5923.73 → 12.7777,168 → 126.81.86×
Multi-document QA7.02K33.23 → 39.1129.90 → 22.4397,888 → 152.61.33×
Few-shot Learning6.55K48.75 → 55.1126.01 → 16.6194,372 → 147.81.57×
Synthetic Tasks9.24K39.16 → 44.3122.89 → 19.1393,977 → 148.81.20×
All Tasks6.70K38.99 → 45.2626.20 → 18.3293,058 → 146.61.43×

Datasets: Single-document QA: Qasper, MultiFieldQA-en; Multi-document QA: HotpotQA, 2WikiMQA; Few-shot Learning: TREC, TriviaQA, SAMSum; Synthetic Tasks: PassageCount, PassageRetrieval-en;

These results extend our evaluation beyond short-context multiple-choice tasks and demonstrate CacheCodec's effectiveness in longer, more diverse downstream workloads.

W3: Writing and formatting

The writing needs polishing. "Concatenation" is repeatedly misspelled as "Concatentation", the DeepSeek-V2 citation appears as "dee, 2024", the main text refers to "Figure A.5" instead of Figure 8, and Appendices A.2.2 and A.3 contain large blank spaces.

All corrections have been incorporated into the latest manuscript.

[1] Bai et al., “LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding,” ACL 2024.

Thank you for the detailed response and the additional experiments.

W1. Thank you for adding the comparison with KIVI, CacheGen and KVQuant. However, the table does not report the payload size or compression ratio of each baseline. Since payload reduction is the core metric of the paper, latency alone does not allow a direct comparison of compression efficiency, and I would suggest adding these numbers.

W2. Thank you for the LongBench-E results. The 6.27 pp improvement over uncompressed C2C is surprisingly large and needs an explanation. Training used a maximum sequence length of 2048, while the evaluation contexts are 4K–9K tokens, so the uncompressed C2C baseline may already degrade beyond its training length. If so, the gain may come from a weaker baseline rather than from compression preserving more useful information.

Payload comparison with KV compression baselines.

W1. Thank you for adding the comparison with KIVI, CacheGen and KVQuant. However, the table does not report the payload size or compression ratio of each baseline. Since payload reduction is the core metric of the paper, latency alone does not allow a direct comparison of compression efficiency, and I would suggest adding these numbers.

Thank you for the suggestion. We agree that E2E latency alone does not reveal the amount of data transmitted. We have added the measured payload size and compression ratio for every evaluated KIVI, CacheGen, and KVQuant configuration, alongside task score and E2E latency. All payloads are measured from the serialized messages sent over the same C2C communication path on matched samples, including the metadata needed for decoding. We define the compression ratio as the total Raw KV bytes divided by the total transmitted bytes for each model pair. As shown in the added table, the evaluated baselines achieve average compression ratios of 2.28–5.03×. Together with the measured payload sizes, these results provide a direct comparison of their communication costs and compression efficiency under matched evaluation conditions.

Receiver for all Sharers: Qwen3-0.6B.

SharerMethodAvg. CR ↑Avg. payload (bytes/sample) ↓
Qwen2.5-0.5B-InstructRaw KV1.00×1,795,837
KIVI-22.99×598,424
KIVI-42.28×786,050
CacheGen2.37×753,750
KVQuant (2-bit)3.93×457,867
KVQuant (4-bit)2.63×682,369
Qwen3-4B-BaseRaw KV1.00×21,550,044
KIVI-23.09×6,950,733
KIVI-42.33×9,201,999
CacheGen2.73×7,868,735
KVQuant (2-bit)5.03×4,293,623
KVQuant (4-bit)3.09×6,987,390
Llama-3.2-1B-InstructRaw KV1.00×4,428,451
KIVI-23.00×1,469,557
KIVI-42.29×1,927,056
CacheGen2.45×1,796,482
KVQuant (2-bit)4.75×932,926
KVQuant (4-bit)2.98×1,486,483

Clarification of training lengths and the LongBench-E gain.

W2. Thank you for the LongBench-E results. The 6.27 pp improvement over uncompressed C2C is surprisingly large and needs an explanation. Training used a maximum sequence length of 2048, while the evaluation contexts are 4K–9K tokens, so the uncompressed C2C baseline may already degrade beyond its training length. If so, the gain may come from a weaker baseline rather than from compression preserving more useful information.

Thank you for raising this concern. The 2,048-token maximum sequence length reported in the paper applies to training on OpenHermes 2.5 for the short-context benchmarks; it does not apply to the LongBench-E experiments. For LongBench-E, both Raw C2C and CacheCodec were trained with the same max_length = 8192 setting. We have clarified the separate training and evaluation configurations in Appendix A.2.

CacheCodec learns a compact, task-oriented representation of the Sharer cache under the Receiver's downstream task objective, rather than optimizing for raw-cache reconstruction. The cache is encoded and decoded before the C2C fuser; the decoded representation then follows C2C integration and downstream generation procedure as the uncompressed baseline. Thus, the comparison does not give CacheCodec a longer training context or a different integration mechanism. Moreover, the improvement is observed across single-document QA, multi-document QA, and few-shot tasks, rather than being confined to one task category. Under the matched C2C setting, the gains across task categories support the effectiveness of our task-oriented training in producing a compact cache representation that better serves the Receiver’s downstream tasks.

Reviewer_YrrHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes CacheCodec, a KV compression way for bandwidth-constrained communication between multi-LLM. It transforms Sharer's KV cache to low-rate bitstream by using a learned Latent Channel Compressor (LCC) and frequency-dependent encoder, which contain a sequence-domain DCT, quantization, and entropy coding. When transferred to Receiver, its decoded representation supports cache concatenation and C2C fusion. The experiment result shows that, across different Sharer models and downstream tasks, CacheCodec reduces transmitted payload by 792x and achieves 8.64x end-to-end latency speed up under a 50 Mbps link, while maintaining or improving Receiver's task accuracy.

Critical strengths
  1. The design of CacheCodec is clear and novel supported by useful component and system evaluations. The paper gives a clear design to both high-dimensional feature extraction and quantization. The paper also train Cachecodec in a task-oriented way: use Receiver-side task loss to represent the quality of compression, rather than align Receiver's KV Cache.
  2. The results of the paper are impressive. The results show substantial payload reductions and lower end-to-end latency than raw KV communication under bandwidth constraints. Average task accuracy also improves. Relative to T2T, CacheCodec offers favorable accuracy–latency trade-offs in several settings.
Critical weaknesses
  1. The current ablation study provides insufficient evidence to justify the structural design choices of the Frequency Codec module. Specifically, the authors do not adequately isolate or validate the contribution of each component in the Frequency Codec, including DCT, quantization, and entropy coding. As a result, it remains unclear whether all of these components are necessary and how much each of them contributes to the overall performance.
  2. The main experimental results lack several key baselines needed to support the authors' claim. In Lines 73–75, the authors argue that existing KV cache compression methods do not consider what information should be preserved for the receiver. However, in Section 4.2, CacheCodec is compared with only one KV communication method, C2C, while no existing KV cache compression methods are included as baselines. I recommend adding baselines that directly integrate representative KV cache compression methods into KV communication pipelines to provide stronger evidence for the claim in Lines 73–75.
Questions for authors
  1. What is the performance of CacheCodec with different Receiver models and different evaluated tasks rather than multiple-choice benchmarks?
  2. What is the distribution of shared sequence lengths and accuracy, payload size, latency change?
  3. What is the performance of CacheCodec on real multi-agent benchmarks?
2 comments

We sincerely thank Reviewer YrrH for the constructive feedback. We address all concerns and questions below with additional experiments and detailed analyses.

W1: Component-wise Ablation of the Frequency Codec

The current ablation study provides insufficient evidence to justify the structural design choices of the Frequency Codec module. Specifically, the authors do not adequately isolate or validate the contribution of each component in the Frequency Codec, including DCT, quantization, and entropy coding. As a result, it remains unclear whether all of these components are necessary and how much each of them contributes to the overall performance.

The component-wise ablations show that DCT and quantization each contribute to both accuracy and compression, while entropy coding further reduces payload without affecting accuracy. To better justify the design of the Frequency Codec, we conducted additional component-wise ablations to examine the individual contributions of DCT, quantization and entropy coding.

Frequency Codec ConfigurationMMLU-ReduxARC-COpenBookQAC-EvalAvg. Acc.Avg. CR
w/o DCT40.152.146.239.744.53636.3×
w/o Quant44.454.046.840.446.404.28x
w/o Entropy Coding44.855.349.641.947.907.95×
CacheCodec44.855.349.641.947.90871.1×

As shown in the table, removing DCT reduces average accuracy by 3.37 pp, while removing quantization leads to a 1.50 pp drop with compression ratios dropping from 871.1× to 4.28×. Removing zlib leaves accuracy unchanged but reduces the compression ratio from 871.1× to 7.95×.

These results support the full Frequency Codec design, which achieves the highest compression ratio among the tested configurations while matching the best average accuracy.

W2: KV Compression Baselines

The main experimental results lack several key baselines needed to support the authors' claim. In Lines 73–75, the authors argue that existing KV cache compression methods do not consider what information should be preserved for the receiver. However, in Section 4.2, CacheCodec is compared with only one KV communication method, C2C, while no existing KV cache compression methods are included as baselines. I recommend adding baselines that directly integrate representative KV cache compression methods into KV communication pipelines to provide stronger evidence for the claim in Lines 73–75.

CacheCodec achieves higher average accuracy and lower average end-to-end latency than KIVI, CacheGen, and KVQuant across all three evaluated model pairs. We additionally evaluate three representative KV-cache compressors: KIVI, CacheGen, and KVQuant across five compression configurations within the same C2C communication pipeline. All methods compress the same Sharer KV cache, transmit the actual serialized payload (including decoding metadata), and use Receiver-side C2C module. We maintain identical evaluation samples, decoding settings, bandwidth, and latency accounting to ensure a fair comparison. Accuracy and E2E latency are averaged equally across four benchmarks: ARC-C, MMLU-Redux, OpenBookQA, and C-Eval.

SharerMethodAvg. Acc. (%) ↑ΔAcc. (pp)Avg. E2E (ms) ↓Speedup ↑
Qwen2.5-0.5B-InstructUncompressed C2C46.88–448.01.00×
KIVI (K2V2)46.95+0.08275.01.63×
KIVI (K4V4)46.78−0.10306.51.46×
CacheGen46.70−0.18324.51.38×
KVQuant (2-bit)46.53−0.35272.31.65×
KVQuant (4-bit)46.75−0.13306.51.46×
CacheCodec48.18+1.30267.51.67×
Llama-3.2-1BUncompressed C2C45.90–1140.81.00×
KIVI (K2V2)46.00+0.10404.02.82×
KIVI (K4V4)45.95+0.05477.02.39×
CacheGen45.88−0.03470.32.43×
KVQuant (2-bit)45.73−0.18330.33.45×
KVQuant (4-bit)45.88−0.03420.02.72×
CacheCodec47.90+2.00164.86.92×
Qwen3-4B-BaseUncompressed C2C44.40–3751.01.00×
KIVI (K2V2)44.23−0.181295.02.90×
KIVI (K4V4)44.38−0.031655.52.27×
CacheGen43.90−0.501467.82.56×
KVQuant (2-bit)44.83+0.43909.04.13×
KVQuant (4-bit)44.10−0.301343.02.79×
CacheCodec46.40+2.00217.317.27×

As shown in the table, existing compressors achieve 1.38–4.13× E2E speedups while maintaining accuracy within −0.50 to +0.43 percentage points (pp) of uncompressed C2C. In contrast, CacheCodec achieves the highest average accuracy and lowest average E2E latency across all three model pairs, with accuracy gains of +1.30 to +2.00 pp and speedups of 1.67–17.27×. Across all 12 model-pair–benchmark settings, CacheCodec improves average accuracy by +1.77 pp and achieves an 8.64× E2E speedup over uncompressed C2C.

By optimizing the transmitted representation using the Receiver’s downstream task objective, CacheCodec achieves both higher accuracy and lower end-to-end latency in heterogeneous communication.

Q1: Generalization to across Receivers and tasks

Q1: What is the performance of CacheCodec with different Receiver models and different evaluated tasks rather than multiple-choice benchmarks?

CacheCodec achieves better task performance and lower end-to-end latency across both evaluated Receiver models and diverse tasks beyond multiple-choice benchmarks.

Sharer → Receiver: Qwen3-0.6B → Qwen2.5-0.5B-Instruct

MethodMMLU-RARC-COpenBookQAC-EvalAvg. Acc. ↑Avg. E2E (ms) ↓Speedup ↑
Uncompressed C2C36.540.742.238.139.38538.81.00×
CacheCodec38.141.942.839.840.65 (+1.28 pp)174.03.10×

Generalization to Different Receiver Models. To evaluate CacheCodec with different Receivers, we swap the Sharer and Receiver roles between Qwen2.5-0.5B-Instruct and Qwen3-0.6B. With Qwen3-0.6B as the Receiver, CacheCodec improves average accuracy by 1.30 percentage points (pp) and achieves a 1.67× E2E speedup over uncompressed C2C. When reversing the direction and using Qwen2.5-0.5B-Instruct as the Receiver, CacheCodec consistently improves accuracy across all four benchmarks (+0.6 to +1.7 pp, +1.28 pp on average) while achieving a 3.10× E2E speedup.

Generalization Across Tasks. CacheCodec focuses on the KV communication interface between heterogeneous LLMs rather than agent coordination strategies. To further evaluate its effectiveness, we extend our experiments to LongBench-E[1], covering single-/multi-document QA, few-shot learning, and synthetic tasks, using their respective task-specific metrics.

These results demonstrate that CacheCodec effectively conveys task-relevant information across diverse downstream tasks, improving the average task score by 6.27 pp (38.99 → 45.26), reducing communication payload by 634.8× (93,058 → 146.6 KB), and achieving a 1.43× E2E speedup (26.20 → 18.32 s). Although LongBench-E[1] does not directly evaluate multi-turn agent coordination, it validates the underlying communication mechanism in heterogeneous Sharer–Receiver settings.

Values are reported as Uncompressed C2C → CacheCodec.

Task TypeAvg. LengthTask Score ↑E2E (s) ↓Payload (KB) ↓Speedup ↑
Single-document QA4.09K29.96 → 37.5923.73 → 12.7777,168 → 126.81.86×
Multi-document QA7.02K33.23 → 39.1129.90 → 22.4397,888 → 152.61.33×
Few-shot Learning6.55K48.75 → 55.1126.01 → 16.6194,372 → 147.81.57×
Synthetic Tasks9.24K39.16 → 44.3122.89 → 19.1393,977 → 148.81.20×
All Tasks6.70K38.99 → 45.2626.20 → 18.3293,058 → 146.61.43×

Datasets: Single-document QA: Qasper, MultiFieldQA-en; Multi-document QA: HotpotQA, 2WikiMQA; Few-shot Learning: TREC, TriviaQA, SAMSum; Synthetic Tasks: PassageCount, PassageRetrieval-en.

Q2: Performance Across Shared Sequence Lengths.

What is the distribution of shared sequence lengths and accuracy, payload size, latency change?

We conduct a length-stratified evaluation on LongBench-E[1], grouping samples by their lengths (0–4K, 4–8K, and 8K+). CacheCodec improves task scores and reduces payload across all three length ranges, while reducing E2E latency for sequences of 4K+. We report the sample distribution and compare CacheCodec with uncompressed C2C in terms of downstream task scores, transmitted payload, and E2E latency under the same bandwidth setting.

Shared KV LengthSamples (%)Task Score ↑Payload (KB) ↓E2E Latency (s) ↓Speedup ↑
0–4K169 (17.8%)27.33 → 34.1331,540 → 65.220.62 → 21.390.96×
4–8K295 (31.2%)34.63 → 38.8068,900 → 117.328.16 → 21.051.34×
8K+483 (51.0%)29.38 → 31.69132,010 → 195.539.69 → 22.391.77×
All947 (100%)30.65 → 34.3494,420 → 147.932.69 → 21.801.50×

As shown in the table, CacheCodec improves task scores by 6.80, 4.17, and 2.31 pp across the three length ranges, respectively, while achieving an average 638.40× payload compression ratio and 1.50× E2E speedup across these ranges.

Q3: Generalization on multi-agent benchmarks

Q3: What is the performance of CacheCodec on real multi-agent benchmarks?

We appreciate the suggestion to evaluate CacheCodec in a multi-agent workflow. Our experiments focus on the communication step itself. In the new LongBench-E[1] evaluation, we hold the Sharer, Receiver, and C2C fusion mechanism fixed and compare compressed and uncompressed KV transfer. CacheCodec reduces the average payload from 93,058 KB to 146.6 KB while increasing the average task score from 38.99 to 45.26. A dedicated multi-agent evaluation would test whether these per-transfer savings translate into lower communication costs while preserving end-to-end task success under the same collaboration protocol. This system-level evaluation remains future work.

[1] Bai et al., “LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding,” ACL 2024.

Comments on the paper

1 comment

Rebuttal Summary

We thank the reviewers for their constructive feedback. The table below summarizes our responses and corresponding experiments, clarifications, and manuscript revisions for each reviewer’s comments. Detailed responses are available in the discussion threads.

ReviewerConcernResponse summary
YrrHFrequency Codec componentsAdded separate DCT, quantization, and entropy-coding ablations, reporting accuracy and CR.
KV compression baselinesAdded KIVI, CacheGen, and KVQuant comparisons in the same C2C pipeline across three model pairs.
Receiver and task coverageAdded the reversed Receiver pair and LongBench-E evaluation covering QA, few-shot learning, and synthetic tasks.
Shared sequence lengthsReported sample distributions, scores, payloads, and latency for 0–4K, 4–8K, and 8K+.
Real multi-agent benchmarksClarified that the current evaluation concerns individual transfers; complete multi-agent workflow evaluation remains future work.
wK6tCompression baselines and payload reportingAdded five compression configurations with accuracy/E2E results, followed by serialized payload sizes and CRs including decoding metadata.
Long-context and diverse tasksAdded LongBench-E results: average task score 38.99 → 45.26 and E2E latency 26.20 → 18.32 s.
LongBench-E training fairnessClarified that both methods use an 8,192-token training limit and the same C2C integration/generation procedure; the 2,048-token setting applies to short-context training.
Writing and formattingCorrected “Concatenation,” the DeepSeek-V2 citation, figure references, and appendix spacing.
DTdJComponent isolation and direct controlsAdded quantized raw-KV and no-DCT controls, reporting accuracy and actual transmitted bytes.
Communication protocol and T2TClarified source-content overlap, the Receiver-only comparison, Sharer prefill, T2T prompts/message length, and message-generation timing.
Deployment and latency accountingExplained pair-dependent crossover bandwidths, modeled-link timing, GPU synchronization/device transfers, and the omitted Table 2 components.
Broader evaluationAdded the reversed Receiver pair and LongBench-E task coverage.
Training and implementationClarified per-pair/per-integration training, layer-specific learned mappings, fixed codec settings, quantization scale/STE, evaluation splits, and sample filtering.
Joint/Split parameter confoundClarified that communication budget is matched but parameter count is not; reported parameter counts and acknowledged the confound.
Uncertainty across seedsReported three independently trained/evaluated seeds (42/43/44), with benchmark-wise means and standard deviations.
Physical-network validationAdded TCP measurements between two physical hosts at 50 Mbps; E2E speedups are 1.63× on ARC-C and 1.57× on OpenBookQA.
Training costReported trainable parameters, wall-clock time, and GPU-hours for C2C and CacheCodec across three model pairs on 8 A100 GPUs.
Headline scope and CR aggregationSpecified the Fusion-only scope of 8.64× latency speedup and 792× CR, reconciled alternative latency aggregations, and defined benchmark/model-pair CR averaging.
Reporting consistencyStandardized KIVI-2/KIVI-4 naming and corrected the Table 2 label and Appendix A.3/A.4 reference.
Transfer and streaming limitationsClarified that separately trained LongBench-E codecs do not establish zero-shot transfer and that serialized bitstreams do not demonstrate incremental streaming.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.