- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents KVCMAS, a KV-cache sharing framework built upon KVComm for KV sharing on multi-agent systems. It improves KVComm by compressing its online states with low rank representations and by performing corrections between consecutive agents to avoid additional prefill. Experiments show improved memory efficiency and serving latency.
Critical strengths
- The paper is well-structured and easy to follow.
- The evaluation is comprehensive, covering multiple workloads and comparisons with a broad set of baselines.
Critical weaknesses
-
The novelty is unclear. KVCMAS is built directly upon KVComm. The main contributions, including low-rank compression for correction and replacing a separate reference cache with the preceding agent’s cache, are relatively straightforward. The paper should more clearly specify the technical novelty beyond combining these components within the existing KVComm framework.
-
Limited evaluation of multi-agent workflows. The evaluation mainly uses a fixed multi-agent structure with fixed agent roles. Although the paper claims support for different workflows, these are not sufficiently evaluated. It remains unclear whether KVCMAS generalizes to more dynamic agentic workflows with runtime subagent spawn/join, agent role variation, or modified communication structures.
-
The paper mainly evaluates its advantage over KV sharing methods to reducing the memory footprint, but does not compare against standard KV memory-management techniques such as quantization or CPU offloading. These techniques could also reduce GPU memory bottleneck without affecting the accuracy. It is unclear whether the reported memory reduction translates into a end-to-end performance advantage.
-
Figure 5 uses different default anchor-pool sizes for KVComm (V=20) and KVCMAS (V=10), while the matched-V is only provided in the appendix. Since KVComm is the closest baseline and is OOM in long-context settings, a direct end-to-end comparison under matched memory configurations would strengthen the evaluation.
-
The overhead of KVCMAS is insufficiently evaluated. Operations such as anchor matching, low-rank reconstruction, and SVD introduce additional computation, but their costs are not separately reported. A breakdown of these overheads would better demonstrate the actual system benefit. The paper should also clarify how the low-rank KV states are stored and reconstructed in the KV pool, and whether this affects the performance of standard FlashAttention kernels. A breakdown of these overheads and details would better demonstrate the actual system benefit.
Questions for authors
Please address the questions raised in the weaknesses above. I would only consider changing my rating if these concerns are fundamentally resolved.
8 comments
Response to Weakness 3
- Comparison with quantization and CPU offloading:
We would like to highlight that quantization and offloading are orthogonal to KVCMAS, as they reduce memory along different axes. KVCMAS reduces the number of KV caches, by sharing one base cache and storing only the agent-specific corrections in low rank. Quantization instead reduces the bit width of each KV cache entry, and offloading moves the entries to CPU memory. Therefore, they can be applied directly on top of KVCMAS, by quantizing or offloading its shared base cache and corrected caches.
However, quantization and offloading alone reduce only memory, not TTFT. They operate on the KV cache after each agent has prefilled the shared context, so every agent still repeats this prefill at its transition. KV cache sharing removes the repeated prefill itself, and thus reduces both TTFT and memory.
We acknowledge that results combining quantization with KV cache sharing would further clarify the impact of our work. We therefore report the accuracy, memory, and TTFT of NonShared and KVCMAS with and without KV cache quantization (KIVI [1]) in Table B. In agentic workloads, where errors accumulate across agents, KV cache quantization leads to a larger accuracy loss while leaving TTFT unchanged. Its memory reduction still applies to KVCMAS, so KVCMAS + KIVI achieves the lowest memory. We updated this clarification in our updated manuscript as well.
Table B. Accuracy (%), peak memory (GB), and TTFT (s) with KIVI 2-bit KV cache quantization, using the setting of the original paper (group size 32, residual length 128). Memory and TTFT are measured under the 32K single-stream setting. Quantization reduces the KV cache memory, while the prefill computation and thus TTFT remain unchanged.
| Method | MMLU | Video-MME | Memory | TTFT |
|---|---|---|---|---|
| NonShared | 67.54 | 53.00 | 31.5 | 8.14 |
| NonShared + KIVI | 63.83 | 52.00 | 18.2 | 8.14 |
| KVCMAS | 65.45 | 52.36 | 29.2 | 4.02 |
| KVCMAS + KIVI | 61.32 | 50.95 | 17.7 | 4.02 |
[1] Liu et al., KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache, ICML 2024.
Response to Weakness 4
- Why the anchor pool sizes differ & Comparison at matched configurations:
We used $ V = 20 $ for KVComm in Figure 5 and the main experiments because its original paper identifies it as its accuracy-efficiency balance, and reducing it to $ V = 10 $ drops its MMLU accuracy by 0.55 points (62.57 to 62.02). However, we agree that a direct comparison under matched configurations would further clarify the impact of our work. We therefore compare KVComm at $ V = 1 $ and $ V = 10 $ with KVCMAS at $ V = 10 $ in Table C, reporting the MMLU accuracy and memory.
At the matched anchor pool capacity ($ V = 10 $), KVCMAS achieves 3.43 points higher MMLU accuracy than KVComm, while KVComm uses $ 2.14\times $ the memory. To match the memory of KVCMAS, KVComm has to reduce its pool to a single anchor ($ V = 1 $), which still uses more memory than KVCMAS (18.18 GB versus 16.06 GB), while its MMLU accuracy is 5.30 points lower. Hence, KVCMAS is both more accurate and more efficient under both the same anchor pool capacity and the same memory budget.
Table C. KVComm and KVCMAS at matched anchor pool capacity and matched memory. Accuracy is in % and memory (GB) is the peak memory on MMLU.
| Method | $ V $ | MMLU accuracy | Memory |
|---|---|---|---|
| KVComm | 1 | 60.15 | 18.18 |
| KVComm | 10 | 62.02 | 34.39 |
| KVCMAS | 10 | 65.45 | 16.06 |
Response to Weakness 5
- Overhead breakdown:
We agree that a breakdown of the additional computation would further clarify our work, and we report the cost of each KVCMAS component in Table D. Anchor matching and low-rank reconstruction remain small compared with the dense prefill they replace. SVD factorization runs after the prefill with first token generation, so it is excluded from TTFT by definition, but it is always included in E2E latency, throughput, and peak memory (Appendix B.1). We updated this clarification in our updated manuscript as well.
Table D. Per-request cost (s) of KVCMAS components under single-stream execution at $ \rho = 0.8 $, compared with the dense prefill.
| Component | 2K | 8K | 32K |
|---|---|---|---|
| Anchor matching and reliability gate | 0.06 | 0.09 | 0.30 |
| Low-rank reconstruction and materialization | 0.10 | 0.09 | 0.10 |
| SVD factorization after the dense prefill | 0.04 | 0.19 | 0.52 |
| Dense prefill (reference) | 0.28 | 1.05 | 6.92 |
- Storage and reconstruction of the low-rank components, and FlashAttention compatibility:
The low-rank components are stored only in the anchor pool, separately from the KV cache used by attention. When a correction is accepted, KVCMAS reconstructs it from the low-rank components, adds it to the source cache, and writes the result as an ordinary full-dimensional KV cache. Therefore, the low-rank components do not affect standard FlashAttention kernels, since attention always runs on the full-dimensional KV cache without any modification. We note that this reconstruction is applied only once at each agent transition, not at every decoding step, so its overhead is marginal. We updated Section 3.3 and Appendix A.3 to describe this more explicitly.
Thank you for your constructive comments. We appreciate the insightful questions and feedback you have provided.
We would like to highlight that KVCMAS addresses the critical overheads of KVComm in both memory and latency, as well as the error propagated by its non-chained delta correction. These overheads already make KVComm impractical at 2K, where it uses about $ 2.5\times $ the memory of the recomputation baselines (Table 14), and it runs out of memory beyond that (Figure 5). Our novelty is twofold, and both parts are critical, such that neither part alone achieves the best accuracy and efficiency. The appendix evaluates chained correction (Appendix B.5) and the rank ablation (Appendix C.2), and Table A combines and extends these results into a direct breakdown.
Low-rank anchor pool: We find that not only the base caches but also the delta corrections between agents are low-rank (effective rank below 32, Figure 2), so the anchor pool can be stored in low rank. As a result, KVCMAS reduces the memory by $ 8.06\times $ compared with the full-rank pool, while staying within about 1 point of its accuracy (Figure 13 and Table A).
Chained correction: We correct each agent from the cache of the preceding agent, which keeps the first agent exact and removes the extra reference prefill. This lowers the median TTFT by 34% at 32K and 8 QPS (Figure 6) and prevents error propagation, without which, for example, MMLU accuracy drops by 6.44 points (Table 10).
Table A. Breakdown of low-rank compression and chained correction at $ V = 10 $. Accuracy is in %, peak memory (GB) is measured under the 32K single-stream setting, and TTFT (s) is the median under concurrent serving at 32K and 8 QPS. Full-rank memory is conservatively computed following Table 14.
To the best of our knowledge, we are the first to identify and clarify these observations, supported by both empirical and theoretical analysis, together with more comprehensive evaluations under diverse conditions, including VLM benchmarks, single-stream execution and concurrent serving, and ablation studies.
In existing multi-agent benchmarks, agent roles are assigned before execution, and dynamic graphs are rarely exercised, and our baselines are also evaluated on such role-specific benchmarks. Within this scope, we evaluated KVCMAS as broadly as possible in both accuracy and efficiency: five workloads with different role sets including VLM benchmarks, varying numbers of agents and rounds (Appendices C.5 and C.6), and both single-stream execution and concurrent serving.
Since KVCMAS itself does not assume a fixed workflow and can be operated on arbitrary agent transitions (Section 2.1), we will add results on dynamic workflows as soon as a suitable benchmark becomes available.