acceptodds
Under review as a conference paper at ICLR 2027

KVCMAS: Efficient KV cache Correction for Shared Context in Multi-Agent Systems

Abstract

Prompt-specialized multi-agent systems enable multiple agents to share a model while performing complementary roles to solve complex tasks. However, agent-specific prefixes change the KV cache generated for the same shared context, causing each agent to repeatedly prefill the growing context and construct a separate cache with high computation and memory overhead. Selective recomputation reduces this redundancy but still retains substantial model execution, while existing delta correction methods either support only recurring context relations or maintain memory-intensive online correction states for dynamically changing context. For first seen shared context, these methods also construct a reference cache outside the agent workflow, and an approximate correction at the first agent affects the outputs passed to subsequent agents. We present **KVCMAS**, an online KV cache correction framework that represents cross-agent cache deviations using compact low-rank states and seamlessly chains corrections along the agent workflow without an additional reference prefill. This design supports dynamically changing shared context while preserving an exact first-agent cache. Across multiple language and vision-language workloads, KVCMAS matches or improves the accuracy of prior KV cache sharing methods while achieving the lowest TTFT under highly concurrent serving. Under controlled serving traces, it provides a 2.0 TTFT speedup over inference without KV cache sharing and reduces peak GPU memory by up to 3.7 relative to a prior KV cache correction method. These results establish KVCMAS as an accurate and scalable KV cache sharing approach for prompt-specialized multi-agent serving.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM, VLM, Agentic AI, Multi-agent, KV cache, Low-rank compression, Agent workflow, Efficient serving, Efficient inference

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_pPqgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents KVCMAS, a KV-cache sharing framework built upon KVComm for KV sharing on multi-agent systems. It improves KVComm by compressing its online states with low rank representations and by performing corrections between consecutive agents to avoid additional prefill. Experiments show improved memory efficiency and serving latency.

Critical strengths
  1. The paper is well-structured and easy to follow.
  2. The evaluation is comprehensive, covering multiple workloads and comparisons with a broad set of baselines.
Critical weaknesses
  1. The novelty is unclear. KVCMAS is built directly upon KVComm. The main contributions, including low-rank compression for correction and replacing a separate reference cache with the preceding agent’s cache, are relatively straightforward. The paper should more clearly specify the technical novelty beyond combining these components within the existing KVComm framework.

  2. Limited evaluation of multi-agent workflows. The evaluation mainly uses a fixed multi-agent structure with fixed agent roles. Although the paper claims support for different workflows, these are not sufficiently evaluated. It remains unclear whether KVCMAS generalizes to more dynamic agentic workflows with runtime subagent spawn/join, agent role variation, or modified communication structures.

  3. The paper mainly evaluates its advantage over KV sharing methods to reducing the memory footprint, but does not compare against standard KV memory-management techniques such as quantization or CPU offloading. These techniques could also reduce GPU memory bottleneck without affecting the accuracy. It is unclear whether the reported memory reduction translates into a end-to-end performance advantage.

  4. Figure 5 uses different default anchor-pool sizes for KVComm (V=20) and KVCMAS (V=10), while the matched-V is only provided in the appendix. Since KVComm is the closest baseline and is OOM in long-context settings, a direct end-to-end comparison under matched memory configurations would strengthen the evaluation.

  5. The overhead of KVCMAS is insufficiently evaluated. Operations such as anchor matching, low-rank reconstruction, and SVD introduce additional computation, but their costs are not separately reported. A breakdown of these overheads would better demonstrate the actual system benefit. The paper should also clarify how the low-rank KV states are stored and reconstructed in the KV pool, and whether this affects the performance of standard FlashAttention kernels. A breakdown of these overheads and details would better demonstrate the actual system benefit.

Questions for authors

Please address the questions raised in the weaknesses above. I would only consider changing my rating if these concerns are fundamentally resolved.

8 comments

Thank you for your constructive comments. We appreciate the insightful questions and feedback you have provided.

Response to Weakness 1

  • Technical novelty beyond KVComm:

We would like to highlight that KVCMAS addresses the critical overheads of KVComm in both memory and latency, as well as the error propagated by its non-chained delta correction. These overheads already make KVComm impractical at 2K, where it uses about $ 2.5\times $ the memory of the recomputation baselines (Table 14), and it runs out of memory beyond that (Figure 5). Our novelty is twofold, and both parts are critical, such that neither part alone achieves the best accuracy and efficiency. The appendix evaluates chained correction (Appendix B.5) and the rank ablation (Appendix C.2), and Table A combines and extends these results into a direct breakdown.

  1. Low-rank anchor pool: We find that not only the base caches but also the delta corrections between agents are low-rank (effective rank below 32, Figure 2), so the anchor pool can be stored in low rank. As a result, KVCMAS reduces the memory by $ 8.06\times $ compared with the full-rank pool, while staying within about 1 point of its accuracy (Figure 13 and Table A).

  2. Chained correction: We correct each agent from the cache of the preceding agent, which keeps the first agent exact and removes the extra reference prefill. This lowers the median TTFT by 34% at 32K and 8 QPS (Figure 6) and prevents error propagation, without which, for example, MMLU accuracy drops by 6.44 points (Table 10).

Table A. Breakdown of low-rank compression and chained correction at $ V = 10 $. Accuracy is in %, peak memory (GB) is measured under the 32K single-stream setting, and TTFT (s) is the median under concurrent serving at 32K and 8 QPS. Full-rank memory is conservatively computed following Table 14.

Low-rankChainedMMLUVideo-MMEPeak MemoryTTFT (s)
✗✗ (KVComm)62.0251.23235.60OOM
✓✗59.0150.8329.2493.86
✗✓66.4852.84235.60OOM
✓✓ (KVCMAS)65.4552.3629.2461.95

To the best of our knowledge, we are the first to identify and clarify these observations, supported by both empirical and theoretical analysis, together with more comprehensive evaluations under diverse conditions, including VLM benchmarks, single-stream execution and concurrent serving, and ablation studies.

Response to Weakness 2

  • Dynamic multi-agent workflows:

In existing multi-agent benchmarks, agent roles are assigned before execution, and dynamic graphs are rarely exercised, and our baselines are also evaluated on such role-specific benchmarks. Within this scope, we evaluated KVCMAS as broadly as possible in both accuracy and efficiency: five workloads with different role sets including VLM benchmarks, varying numbers of agents and rounds (Appendices C.5 and C.6), and both single-stream execution and concurrent serving.

Since KVCMAS itself does not assume a fixed workflow and can be operated on arbitrary agent transitions (Section 2.1), we will add results on dynamic workflows as soon as a suitable benchmark becomes available.

Response to Weakness 3

  • Comparison with quantization and CPU offloading:

We would like to highlight that quantization and offloading are orthogonal to KVCMAS, as they reduce memory along different axes. KVCMAS reduces the number of KV caches, by sharing one base cache and storing only the agent-specific corrections in low rank. Quantization instead reduces the bit width of each KV cache entry, and offloading moves the entries to CPU memory. Therefore, they can be applied directly on top of KVCMAS, by quantizing or offloading its shared base cache and corrected caches.

However, quantization and offloading alone reduce only memory, not TTFT. They operate on the KV cache after each agent has prefilled the shared context, so every agent still repeats this prefill at its transition. KV cache sharing removes the repeated prefill itself, and thus reduces both TTFT and memory.

We acknowledge that results combining quantization with KV cache sharing would further clarify the impact of our work. We therefore report the accuracy, memory, and TTFT of NonShared and KVCMAS with and without KV cache quantization (KIVI [1]) in Table B. In agentic workloads, where errors accumulate across agents, KV cache quantization leads to a larger accuracy loss while leaving TTFT unchanged. Its memory reduction still applies to KVCMAS, so KVCMAS + KIVI achieves the lowest memory. We updated this clarification in our updated manuscript as well.

Table B. Accuracy (%), peak memory (GB), and TTFT (s) with KIVI 2-bit KV cache quantization, using the setting of the original paper (group size 32, residual length 128). Memory and TTFT are measured under the 32K single-stream setting. Quantization reduces the KV cache memory, while the prefill computation and thus TTFT remain unchanged.

MethodMMLUVideo-MMEMemoryTTFT
NonShared67.5453.0031.58.14
NonShared + KIVI63.8352.0018.28.14
KVCMAS65.4552.3629.24.02
KVCMAS + KIVI61.3250.9517.74.02

[1] Liu et al., KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache, ICML 2024.

Response to Weakness 4

  • Why the anchor pool sizes differ & Comparison at matched configurations:

We used $ V = 20 $ for KVComm in Figure 5 and the main experiments because its original paper identifies it as its accuracy-efficiency balance, and reducing it to $ V = 10 $ drops its MMLU accuracy by 0.55 points (62.57 to 62.02). However, we agree that a direct comparison under matched configurations would further clarify the impact of our work. We therefore compare KVComm at $ V = 1 $ and $ V = 10 $ with KVCMAS at $ V = 10 $ in Table C, reporting the MMLU accuracy and memory.

At the matched anchor pool capacity ($ V = 10 $), KVCMAS achieves 3.43 points higher MMLU accuracy than KVComm, while KVComm uses $ 2.14\times $ the memory. To match the memory of KVCMAS, KVComm has to reduce its pool to a single anchor ($ V = 1 $), which still uses more memory than KVCMAS (18.18 GB versus 16.06 GB), while its MMLU accuracy is 5.30 points lower. Hence, KVCMAS is both more accurate and more efficient under both the same anchor pool capacity and the same memory budget.

Table C. KVComm and KVCMAS at matched anchor pool capacity and matched memory. Accuracy is in % and memory (GB) is the peak memory on MMLU.

Method$ V $MMLU accuracyMemory
KVComm160.1518.18
KVComm1062.0234.39
KVCMAS1065.4516.06

Response to Weakness 5

  • Overhead breakdown:

We agree that a breakdown of the additional computation would further clarify our work, and we report the cost of each KVCMAS component in Table D. Anchor matching and low-rank reconstruction remain small compared with the dense prefill they replace. SVD factorization runs after the prefill with first token generation, so it is excluded from TTFT by definition, but it is always included in E2E latency, throughput, and peak memory (Appendix B.1). We updated this clarification in our updated manuscript as well.

Table D. Per-request cost (s) of KVCMAS components under single-stream execution at $ \rho = 0.8 $, compared with the dense prefill.

Component2K8K32K
Anchor matching and reliability gate0.060.090.30
Low-rank reconstruction and materialization0.100.090.10
SVD factorization after the dense prefill0.040.190.52
Dense prefill (reference)0.281.056.92
  • Storage and reconstruction of the low-rank components, and FlashAttention compatibility:

The low-rank components are stored only in the anchor pool, separately from the KV cache used by attention. When a correction is accepted, KVCMAS reconstructs it from the low-rank components, adds it to the source cache, and writes the result as an ordinary full-dimensional KV cache. Therefore, the low-rank components do not affect standard FlashAttention kernels, since attention always runs on the full-dimensional KV cache without any modification. We note that this reconstruction is applied only once at each agent transition, not at every decoding step, so its overhead is marginal. We updated Section 3.3 and Appendix A.3 to describe this more explicitly.

Reviewer_J1JHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

KVCMAS proposes an online KV-cache correction framework for prompt-specialized multi-agent systems. Building upon KVComm, it introduces low-rank anchor compression and chained cache referencing to reduce memory overhead and redundant prefilling. Experiments across five language and vision-language benchmarks show accuracy broadly comparable to KVComm, with up to 2.0× TTFT improvement over non-shared inference and 3.7× peak GPU memory reduction relative to prior correction methods.

Critical strengths

S1. Practical improvements in memory efficiency

KVCMAS substantially reduces the memory overhead of KVComm through low-rank anchor compression. The reported memory savings are meaningful, particularly for memory-constrained multi-agent serving.

S2. Effective reference reuse across agents

Chained correction removes the need for a separate reference prefill while preserving an exact cache for the first agent. The experiments demonstrate improvements in both accuracy and TTFT over the non-chained variant.

Critical weaknesses

W1. Limited methodological novelty over KVComm

The proposed framework appears largely incremental to KVComm. Online anchor matching, similarity-weighted correction, and reliability gating are inherited from prior work. The two primary modifications—truncated-SVD-based anchor compression and chained reference selection—are relatively straightforward extensions rather than fundamentally new correction mechanisms.

Although the reported improvements are practically meaningful, the paper does not adequately establish whether they require the proposed framework or could be achieved through simple modifications to KVComm. In particular, controlled comparisons against KVComm augmented with low-rank compression, chained referencing, and their combination are missing. This substantially weakens the claimed methodological contribution.

W2. Insufficient validation of accuracy and end-to-end serving benefits

The strongest latency results are obtained with 32K-token shared contexts, whereas the reported accuracy benchmarks involve substantially shorter contexts, with maximum lengths below approximately 9K tokens. Consequently, the paper does not establish whether accuracy is preserved under the same long-context conditions used to demonstrate the headline speedups.

Furthermore, the concurrent-serving evaluation primarily measures TTFT using trace-driven first-token generation rather than complete multi-agent decoding. Such experiments do not fully capture decode contention, memory pressure, and queuing behavior in realistic concurrent deployments. Full-generation concurrent serving results, including end-to-end latency and throughput, are necessary to substantiate the practical benefits.

Questions for authors
  1. Can the authors compare KVCMAS against KVComm augmented with truncated-SVD anchor compression, chained reference selection, and their combination under identical memory budgets and serving configurations? This would help isolate the contributions beyond straightforward extensions of KVComm.

  2. Can the authors report end-task accuracy at 32K shared-context lengths and evaluate full-generation concurrent serving, including end-to-end latency, throughput, and tail latency? These results would clarify whether the reported accuracy–latency trade-offs hold under the conditions where the largest speedups are observed.

5 comments

Thank you for your constructive comments. We appreciate the insightful questions and feedback you have provided.

Response to Weakness 1 and Question 1

  • Novelty of KVCMAS:

We would like to state that the novelty of KVCMAS indeed lies in those two observations and the designs built on them: to the best of our knowledge, we are the first to identify and clarify these observations, supported by both empirical and theoretical analysis, together with more comprehensive evaluations under diverse conditions, including VLM benchmarks, single-stream execution and concurrent serving, and ablation studies.

First, we observe that not only the base cache but also the key and value delta corrections between agents are low-rank, with an effective rank below 32 across workloads (Section 3.1, Figure 2, and Appendix A.2), which enables low-rank compression of the entire anchor pool.

Second, and more importantly, we introduce correcting the shared context along the multi-agent workflow which is equivalent to correcting from a context-free reference. This yields lower reconstruction error with empirical corrections (Eq. 6, Figure 3) and removes the additional reference prefill entirely.

  • Effectiveness of each component:

As a result, KVCMAS is more memory- and latency-efficient than KVComm while maintaining its accuracy, which neither low-rank compression nor chained correction alone achieves. Appendix B.5 reports the accuracy and serving efficiency of chained correction, and Appendix C.2 reports the accuracy and efficiency of the rank ablation. Without chained correction, accuracy drops by 6.44 and 1.53 points on MMLU and Video-MME, while chained correction reduces the median TTFT by 34% at 32K and 8 QPS. In the rank ablation, the low-rank representation stays within about 1 point of the full-rank accuracy with nearly unchanged TTFT and throughput, while reducing peak memory by $ 8.06\times $.

However, we acknowledge that an explicit breakdown of these components would further clarify their impact. We therefore present Table A, which combines and extends the appendix results, and updated our manuscript to include this.

Table A. Breakdown of low-rank compression and chained correction at $ V = 10 $. Accuracy is in %. Peak memory is measured under the 32K single-stream setting, and TTFT is the median under concurrent serving at 32K and 8 QPS. Full-rank memory exceeds the GPU capacity and is conservatively computed following Table 14.

Low-rankChainedMMLU (%)Video-MME (%)Peak Memory (GB)TTFT (s)
✗✗ (KVComm)62.0251.23235.60OOM
✓✗59.0150.8329.2493.86
✗✓66.4852.84235.60OOM
✓✓ (KVCMAS)65.4552.3629.2461.95

Response to Weakness 2 and Question 2 (part 1/2)

  • Accuracy at long shared contexts:

Since benchmarks for long-context multi-agent KV cache sharing are lacking, previous works such as KVComm, RelayCaching, and GraphFlow evaluate accuracy on agentic tasks together with efficiency on controlled long traces. We further extend the evaluation to longer contexts by adding VLM benchmarks, such as Video-MME with 32 frames (about 7K tokens per trajectory), although benchmarks that combine multi-role assignments with long contexts are still difficult to find. Within this range, the gain of KVCMAS is larger on the VLM benchmarks with longer contexts, reaching a $ 2.65\times $ end-to-end latency speedup over NonShared on MathVista (Table 1).

Acknowledging the reviewer's concern, we evaluate Video-MME with 128 frames (about 25K tokens per trajectory), where KVCMAS keeps its accuracy within 1.91 points of NonShared (Table B). We have updated this ablation on the manuscript and are also looking for additional long-context multi-agent benchmarks and will update the results when available.

Table B. Video-MME accuracy (%) with 128 frames (about 25K tokens per trajectory). KVComm is excluded because its full-dimensional anchor pool alone requires about 140 GB, which, together with the model weights and the active KV cache, exceeds the capacity of our single-node server with 4x NVIDIA A6000 48 GB GPUs.

MethodNonSharedFullSharedDroidSpeakCacheBlendRelayCachingKVCommKVCMAS
Accuracy54.1429.3044.5945.9626.11OOM52.23

Response to Weakness 2 and Question 2 (part 2/2)

  • Full-generation concurrent serving:

We note that our controlled traces replay the multi-agent execution and TTFT is the average of each agent response: the shared context is reused at each agent transition, and the agent outputs are accumulated into the next inputs, only with the prefill and generation lengths fixed across methods. Under concurrent serving, we reported TTFT because it is considered the most critical term that is directly perceived by users. We measured E2E latency and throughput separately under single-stream execution (Table 15) to show the effect of the KV cache sharing mechanism itself, without the request scheduling and batching of concurrent serving.

However, we acknowledge that E2E latency under concurrent serving would further clarify the serving benefit of KVCMAS, and we therefore report E2E latency and throughput under concurrent serving in Table C. With full generation on the 32K trace, KVCMAS achieves the best results among the KV cache sharing works under load, with up to $ 6.3\times $ lower p50 E2E latency and up to $ 2.5\times $ higher throughput than NonShared. We added the experiments and explanations to the appendix as well.

Table C. E2E latency and throughput under concurrent serving on the 32K controlled trace, where the decoding time of each request is measured under full generation. TTFT under the same setting is reported in Table 8(c).

Table C(a). Median (p50) end-to-end latency (s).

$ \rho $0.90.80.6
QPS00.52800.52800.528
NonShared5.564.8115.1127.75.564.8115.1127.75.564.8115.1127.7
FullShared1.83.121.033.61.83.121.033.61.83.121.033.6
DroidSpeak5.463.1113.5125.95.462.4112.6124.35.461.9112.4125.1
CacheBlend5.153.4103.9116.55.563.5113.6126.45.563.1113.7126.5
RelayCaching5.154.4104.9117.25.463.3113.6126.35.563.2114.3126.4
GraphFlow2.711.663.176.23.723.974.587.05.541.992.6105.3
KVCMAS2.710.338.150.72.811.851.864.05.534.879.391.9

Table C(b). Tail (p90) end-to-end latency (s).

$ \rho $0.90.80.6
QPS00.52800.52800.528
NonShared5.6112.1202.7225.45.6112.1202.7225.45.6112.1202.7225.4
FullShared4.45.933.957.54.45.933.957.54.45.933.957.5
DroidSpeak5.5109.3199.7222.35.5108.0197.8219.45.5107.0197.8221.2
CacheBlend5.492.2182.9205.65.5109.9200.1223.15.5109.3200.2223.6
RelayCaching5.494.0184.8207.05.5109.5200.1222.95.5109.7201.0223.5
GraphFlow5.515.6105.1128.35.536.8127.2149.65.573.0164.4187.4
KVCMAS5.512.961.484.35.514.487.1109.95.652.4136.9159.7

Table C(c). Throughput (tokens/sec).

$ \rho $0.90.80.6
QPS00.52800.52800.528
NonShared23.235.436.937.223.235.436.937.223.235.436.937.2
FullShared69.782.2129.1132.069.782.2129.1132.069.782.2129.1132.0
DroidSpeak23.635.837.337.723.636.037.638.223.636.137.637.9
CacheBlend25.338.540.240.723.535.737.337.623.335.837.237.5
RelayCaching25.038.239.940.423.535.737.337.623.335.737.137.5
GraphFlow47.858.262.863.734.951.054.255.123.342.044.044.5
KVCMAS47.959.391.793.946.058.672.273.623.246.751.151.8
Reviewer_SabBOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses an efficiency problem in prompt-specialized multi-agent systems: agents share model weights but carry different role prefixes, so the same shared context produces a different KV cache per agent, forcing each to re-prefill the growing context. The paper proposes KVCMAS, an online correction framework representing cross-agent cache deviations with low-rank states and chaining corrections along the agent workflow. It rests on two ideas. First, low-rank anchors: cross-agent deviations have effective rank below 32, so anchor base caches and corrections are stored as truncated SVD factors, cutting anchor memory from O(VNLD) to O(VNr(L+D)). Second, chained correction: rather than correcting from a context-free reference built outside the workflow, each correction is taken relative to the previous agent's cache. Under exact corrections the two references are algebraically identical; under approximation they differ, and chaining keeps the first agent's cache exact while removing the reference prefill prior delta-correction methods need for first-seen context. KVCMAS is evaluated on five workloads against selective-recomputation and delta-correction baselines, reporting a 2.0× TTFT speedup over inference without sharing and up to 3.7× lower peak GPU memory than KVComm.

Critical strengths
  1. The problem is important and the core observation is clean and inspiring. Cross-agent cache divergence is a real and growing cost in multi-agent serving, and the central measurement — that cross-agent deviations carry an effective rank below 32, far under the full feature dimension — is what makes the compression principled rather than heuristic. The 50 GiB worked example for a 4K segment pool makes the problem concrete before the solution arrives, and checks out exactly against the model's dimensions.

  2. Chained correction is a genuinely insightful contribution. Observing that a context-free reference and the preceding agent's cache are algebraically equivalent under exact corrections but diverge once corrections are approximate, then choosing the reference that both keeps the first agent exact and eliminates an out-of-workflow prefill, is the kind of observation that looks obvious only in hindsight. To the best of my knowledge it is new, and Figure 6 isolates it cleanly: a 34% median TTFT reduction at 32K and 8 QPS against the non-chained variant with everything else fixed.

  3. The systems evaluation is thorough. Five workloads spanning text and vision-language models, a vLLM implementation, a QPS sweep reporting both median and tail latency, three runs per accuracy measurement with standard deviations, and ablations over the reliability threshold, rank, anchor pool size, agent count and interaction rounds. This is more complete than the area usually receives, and it is what makes the efficiency claims assessable at all.

Critical weaknesses
  1. Several comparisons are not well controlled, and the headline numbers depend on it. The clearest case is the memory claim. KVCMAS uses V = 10 anchors while KVComm runs at V = 20 (Appendix C.3), and KVComm's anchor memory is linear in V. Every non-KVComm baseline sits at roughly 16 GB, fixing the model and active-cache floor, so KVComm's anchor pool on MMLU is about 36 GB of its 52.40 GB total; at V = 10 that would be about 18 GB, a total near 34 GB and a ratio of roughly 2.1× rather than 3.26×. A meaningful share of the headline memory advantage therefore reflects a larger anchor pool rather than low-rank storage. This needs either an explanation or a re-run at matched V before the 3.7× figure can be read as a property of low-rank storage.

  2. The selective-recomputation baselines deliver almost nothing at 32K, which is difficult to believe. In Table 7(c) at ρ = 0.8 and 8 QPS, NonShared reaches 125.89 s while DroidSpeak, CacheBlend and RelayCaching reach 122.50, 124.64 and 124.55 s — a 1–3% improvement over not sharing the cache at all, at the very scale the paper emphasizes. Three independent prior methods all collapsing to the no-sharing baseline suggests a problem with how they were configured or implemented rather than a property of selective recomputation, and since these are the primary comparison points the result needs investigating before it can be read as evidence for KVCMAS.

  3. End-to-end latency is confounded by output length and is not analyzed. Table 1 reports latency across methods whose accuracy differs by tens of points, and Table 4 shows average trajectory lengths varying by roughly 30% on the same workload — 8,745 tokens for RelayCaching against 6,594 for NonShared on Video-MME. A method that answers incorrectly can generate substantially more or fewer tokens, so these latencies partly measure response length rather than serving cost. The paper acknowledges this in passing and defers to the controlled traces, yet Table 1 still presents latency as a headline comparison without correcting for it.

Questions for authors

Why is KVComm evaluated at V = 20 while KVCMAS uses V = 10? What is the peak-memory comparison at matched V, and how much of the 3.7× reduction survives it?

In Table 7(c), why do DroidSpeak, CacheBlend and RelayCaching improve TTFT by only 1–3% over NonShared at 32K? How were their recomputation budgets implemented, and is attention over the full context dominating the savings?

Can latency be reported in a form that controls for generated-token count, for example per output token or at a fixed output length?

Appendix C.1 estimates KVComm's memory rather than measuring it, since it exceeds the A6000. How sensitive are the 2.5×, 6.2× and 14.9× figures to that estimate?

4 comments

Thank you for your constructive comments. We appreciate the insightful questions and feedback you have provided.

Response to Weakness 1 and Question 1

  • Why KVComm uses $ V = 20 $:

We used $ V = 20 $ for KVComm because its original paper identifies this capacity as its best accuracy-efficiency balance, which accuracy saturates beyond, and we kept it so as not to weaken the baseline. KVCMAS, in contrast, reaches sufficient accuracy at $ V = 10 $. As shown in Table A, reducing the capacity of KVComm to $ V = 10 $ lowers its accuracy, while it still uses $ 2.14\times $ the memory of KVCMAS. We acknowledge that a comparison at matched $ V $ would highlight the memory benefit of our design, and we provide the analysis below, which is added to the revised paper.

  • Peak memory at matched $ V $, and how much of the reduction survives:

We confirm that the KV cache memory of both KVComm and KVCMAS grows linearly with the anchor pool capacity $ V $. We would like to clarify, however, that the memory reduction of KVCMAS comes mainly from its low-rank representation of the anchor states, which reduces KV cache memory by about $ 20\times $ or more, rather than from the choice of $ V $, which accounts for only an additional $ 2\times $ in our experiments. In detail, for a shared context length $ L $, $ N $ agents, and key/value projection dimension $ D $, KVComm stores KV caches of size $ O(VNLD) $ and KVCMAS stores $ O(VNr(L+D)) $ in each pool. At the same $ V $ and $ N $, their ratio is therefore $ LD/(r(L+D)) .ForLlama−3.1−8B(. For Llama-3.1-8B (D = 1024 $) at $ r = 32 $, this ratio is $ 20\times $ at the MMLU context length and approaches the theoretical maximum of $ D/r = 32\times $ as the shared context grows.

Because both methods carry the same model weights, the reduction in total memory is smaller on short contexts and grows with the shared context length: at $ V = 10 $, it is $ 2.14\times $ on MMLU with 2.3K shared tokens and $ 8.06\times $ on the 32K trace (Tables A and B).

Table A. MMLU accuracy (%), total peak GPU memory (GB), and KV cache memory (GB) across anchor pool capacities on an NVIDIA A100 80 GB GPU at a matched reuse ratio. Numbers in parentheses are the KVComm-to-KVCMAS ratio.

$ V $Acc. KVCMASAcc. KVCommTotal Mem. KVCMASTotal Mem. KVCommKV Cache Mem. KVCMASKV Cache Mem. KVComm
162.5160.1515.3818.18 (1.18x)0.353.16 (9.0x)
564.4261.2715.6825.39 (1.62x)0.6510.36 (15.8x)
1065.4562.0216.0634.39 (2.14x)1.0319.36 (18.7x)
2065.8062.5716.8252.40 (3.12x)1.7937.37 (20.9x)

Table B. Total memory usage (GB) under the controlled single-stream trace across anchor pool capacities at a matched reuse ratio. Numbers in parentheses are the KVComm-to-KVCMAS ratio. KVComm values exceed the A6000 48 GB capacity and are conservatively computed following Table 14.

$ V $2K KVCMAS2K KVComm8K KVCMAS8K KVComm32K KVCMAS32K KVComm
116.4918.55 (1.12x)18.0625.90 (1.43x)26.2155.60 (2.12x)
516.7823.55 (1.40x)18.7245.90 (2.45x)26.68135.60 (5.08x)
1017.1329.80 (1.74x)19.5370.90 (3.63x)29.24235.60 (8.06x)
2017.8442.30 (2.37x)21.16120.90 (5.71x)31.94435.60 (13.64x)

Response to Weakness 2 and Question 2

  • Why selective recomputation shows limited gains under concurrent serving:

We clarify that the limited gain of token-wise selective recomputation under concurrent serving is due to its scattered token selection, which is unfavorable to the paged KV cache management of serving systems, rather than to an implementation or configuration issue.

Serving systems such as vLLM and SGLang manage the KV cache in pages, so even a small scattered selection of 10–20% of the tokens accesses and updates over 80% of the cache pages of the shared context, as shown in Table C. As the reviewer pointed out, this effect is amplified at long context because each recomputed token attends over the entire shared context. By comparison, single-stream execution without paged KV cache management lets CacheBlend and RelayCaching provide 1.4–1.6× TTFT speedups over NonShared at 32K (Table 15).

Delta correction, in contrast, updates the corrected cache entries without model execution, so no cache page is recomputed for reused requests. KVCMAS further avoids the additional reference prefill of other delta correction methods through chained correction, and therefore retains its single-stream gain under concurrent serving. We updated these explanations in the revised paper for clarity.

Layer-wise selective recomputation such as DroidSpeak propagates the hidden states of all shared-context tokens through the model to update the selected layers, and thus provides limited speedups under both concurrent serving and single-stream execution, only reducing memory usage. This is stated in Section 4.3.

Table C. TTFT speedup over NonShared at 32K and $ \rho = 0.8 $ under concurrent serving (the setting of Table 8(c) at QPS 0, p50) and single-stream execution (the setting of Table 15), with the recomputed tokens and cache pages of the shared context per reused request.

MethodRecomputed tokens / recomputed cache pagesConcurrentSingle-stream
CacheBlend20% / 97%1.02x1.43x
RelayCaching20% / 98%1.02x1.56x
GraphFlow- (delta correction)1.98x1.71x
KVCMAS- (delta correction)3.69x2.02x

Response to Weakness 3 and Question 3

  • Latency at a fixed output length:

We acknowledge the reviewer's concern and clarify that the latency gain of KVCMAS on the benchmarks comes not only from its higher accuracy, which avoids unnecessarily long generation, but also from the efficiency of its mechanism itself, as shown by the latency per generated token.

As shown in Table D, when normalized by the number of generated tokens, KVCMAS achieves the lowest latency among the correction methods, with up to 1.24× lower latency per generated token and up to 1.62× TTFT speedup over NonShared (Table 1). This efficiency gain is even more pronounced in long-context settings with a fixed output length, where KVCMAS achieves a 1.40× end-to-end throughput gain under single-stream execution (Table 15) and a 2.0× TTFT speedup under concurrent serving at 8 QPS (Table 8) over NonShared at 32K. We added the normalized latency to the appendix of the revised manuscript.

Table D. End-to-end latency normalized by the number of generated tokens per trajectory (ms per token).

MethodMMLUGSM8KHumanEval
NonShared18.4123.0612.92
DroidSpeak20.8819.9213.83
CacheBlend17.9319.5114.89
RelayCaching24.5328.1516.63
GraphFlow18.3422.9612.28
KVComm15.1319.3612.52
KVCMAS15.0018.5411.97

Response to Question 4

  • Sensitivity of the KVComm memory estimate:

The KVComm memory in Appendix C.1 is conservatively computed. It adds the full-dimensional KV caches stored in the anchor pool of KVComm, namely the base cache and delta corrections of each anchor, together with the copies created when they are applied, to the measured memory shared by all methods, such as the model weights and the active KV cache. Specifically, with $ V $ anchors, $ N $ agents, shared context length $ L $, $ \ell $ layers, and key/value projection dimension $ D $, the estimate is $ M_{\text{shared}} + (V(N+1) + 3) \cdot 2\ell L D \cdot 2 $ bytes, where each anchor stores one base cache and $ N $ delta corrections, three full-dimensional copies are created when they are applied, the factor $ 2\ell L D $ counts the key and value entries across layers, and each entry takes 2 bytes.

Since the estimate excludes allocator overhead and memory fragmentation, it is a lower bound on the actual requirement, and the $ 2.5\times $, $ 6.2\times $, and $ 14.9\times $ figures are therefore conservative. We also note that this computation agrees well with the actual memory usage in accuracy benchmark runs.

Thank you for the clarification and additional experiments. I also remain unconvinced that the additional findings establish a sufficiently substantial conceptual advance over prior work. I therefore maintain my original rating.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Global Response to the Reviewers

We sincerely thank all reviewers for their thoughtful feedback and insightful questions. We kindly ask the reviewers to refer to the updated manuscript for all table, figure, section, and appendix numbers in our responses.

Several themes appeared consistently across the reviews, so we address two of the most frequently raised concerns here: (a) the anchor pool capacity of KVComm, and (b) the novelty of KVCMAS and the contribution of each component. We have updated the manuscript to include clearer explanations and extended experimental results.

Anchor Pool Capacity of KVComm

Across multiple reviews, there was concern that KVComm uses a larger anchor pool ($ V = 20 )thanKVCMAS() than KVCMAS ( V = 10 $), which may inflate the reported memory reduction. We used $ V = 20 $ for KVComm, which its original paper identifies as the balance between efficiency and task performance, so as not to weaken the baseline. Furthermore, the memory reduction of KVCMAS comes mainly from its low-rank anchor pool rather than from the choice of $ V $. At the matched $ V = 10 $, KVComm uses $ 18.7\times $ the KV cache memory and $ 2.14\times $ the total memory of KVCMAS on MMLU, while KVCMAS is 3.43 points more accurate. Even with a single anchor ($ V = 1 $), KVComm still uses more memory than KVCMAS (18.18 GB versus 16.06 GB) and is 5.30 points less accurate.

Since both methods carry the same model weights, the gap is smaller on short contexts and grows with the shared context length as the anchor pool dominates the memory: at $ V = 10 $, KVComm requires $ 1.74\times $, $ 3.63\times $, and $ 8.06\times $ the memory of KVCMAS at 2K, 8K, and 32K, respectively.

Table G1: MMLU accuracy, total memory, and KV cache memory on an A100 80 GB GPU at a matched reuse ratio.

Method$ V $MMLU (%)Total Mem. (GB)KV Cache Mem. (GB)
KVComm160.1518.183.16
KVComm1062.0234.3919.36
KVComm2062.5752.4037.37
KVCMAS1065.4516.061.03

While the original submission compared the memory at the matched $ V = 20 $ in Appendix C.3, we acknowledge that the main comparison used the default capacities. We added the comparison at matched capacity and matched memory to the revised paper for better clarity and visibility.

Novelty of KVCMAS and Component Breakdown

Several reviewers asked about the novelty of KVCMAS beyond KVComm, and whether simply adding low-rank compression or chained correction to KVComm would achieve the same improvements.

We would like to clarify that the novelty of KVCMAS lies in two observations and the designs built on them, which, to the best of our knowledge, we are the first to identify:

  1. Low-rank anchor pool: Not only the base caches but also the key and value delta corrections between agents are low-rank, with an effective rank below 32 (Section 3.1, Figure 2, and Appendix A.2), so the entire anchor pool can be stored in low rank.

  2. Chained correction: Correcting each agent from the cache of the preceding agent is equivalent to correcting from a context-free reference (Eq. 6), but yields lower error with estimated corrections (Figure 3), keeps the first agent exact, and removes the additional reference prefill.

The two designs are crucial, and neither alone achieves the accuracy and efficiency of KVCMAS. The table below combines and extends Appendix B.5 and C.2 into a breakdown at $ V = 10 $. Low-rank compression alone reduces the peak memory by $ 8.06\times $, but without chained correction its accuracy is 6.44 and 1.53 points lower than KVCMAS on MMLU and Video-MME. Chained correction alone is the most accurate, but its full-dimensional anchor pool runs out of memory at 32K. KVCMAS keeps the compact memory of the low-rank pool, stays within about 1 point of the full-rank accuracy, and reduces the median TTFT by 34% at 32K and 8 QPS by removing the reference prefill.

Table G2: Breakdown at $ V = 10 $. Accuracy is in %, peak memory is measured under the 32K single-stream setting, and TTFT is the median under concurrent serving at 32K and 8 QPS. Full-rank memory is conservatively computed following Table 14.

Low-rankChainedMMLU (%)Video-MME (%)Peak Memory (GB)TTFT (s)
✗✗ (KVComm)62.0251.23235.60OOM
✓✗59.0150.8329.2493.86
✗✓66.4852.84235.60OOM
✓✓ (KVCMAS)65.4552.3629.2461.95

We acknowledge that the original submission presented these components in separate ablations. We added this breakdown and further explanations to the revised paper for better clarity and visibility.


We thank the reviewers again for their constructive feedback, and hope the concerns are resolved through our responses.

Dear Reviewers,

The authors have posted their rebuttals. As the author-reviewer discussion period is coming to a close soon, please read their responses and engage in the discussion. Thank you for your time and effort.

Best,

AC

We sincerely thank the reviewers for their detailed feedback and the AC for guiding the discussion. We would be grateful if the reviewers could check our responses within the discussion period and let us know if any concern remains. We believe that a few critical gaps may still remain between some of the concerns and our presentation of the actual contributions, and we have clarified them in the discussion. The main points are summarized below.

Major strengths noted by the reviewers: The reviewers found the problem important (SabB) and the evaluation comprehensive across multiple workloads and a broad set of baselines (SabB, pPqg). Reviewer SabB noted that the low-rank observation "is what makes the compression principled rather than heuristic" and described chained correction as "a genuinely insightful contribution" that is new to their knowledge. Reviewer J1JH noted the meaningful memory savings and the accuracy and TTFT gains of chained correction. After the rebuttal, Reviewer J1JH found that the component breakdown results "largely address my concern about the effectiveness of chained correction", and Reviewer pPqg found that the TraceLab and CPU offloading results "address several of my original concerns".

Novelty and Contributions

  • Chained correction (SabB, J1JH, pPqg): To our knowledge, chained correction is the first to change the reference of delta correction, from a separately constructed context-free cache to the cache that the source agent has already produced (Eq. 6). It processes each new shared segment once instead of twice, overwrites each source cache in place, and keeps the first agent's cache exact, so no correction error propagates from it (Figure 3).

  • Beyond combining components with KVComm (J1JH, pPqg): Compressing the anchors of KVComm alone lowers accuracy. Adding chained correction to the compressed anchors raises accuracy above KVComm and reduces the median TTFT by 34% under the same memory (Table 19).

  • Positioning against prior work (J1JH, pPqg): Our low-rank observation concerns the deviation between agents over new shared context, which KVCMAS estimates online. It differs from the low-rank KV caches of ShadowKV and xKV and from the corrections that Kamera forms through a conditioned forward for recurring antecedents. KVCMAS also corrects without model execution, unlike the recomputation of CacheBlend and DroidSpeak, and is more accurate than all recomputation baselines on every workload (Table 1).

Controlled and end-to-end evaluation

  • Matched anchor pool (SabB, pPqg): At the same anchor pool capacity, KVComm uses about twice the memory of KVCMAS and is less accurate. Even with a single anchor, it still uses more memory and is less accurate (Table 18).

  • Recomputation under serving (SabB): Under paged KV cache management, recomputing a small fraction of scattered tokens touches nearly all cache pages, which limits token-wise recomputation under concurrent serving, while the same methods remain effective in single-stream execution (Appendix B.4).

  • Long shared context (J1JH): With the longest videos in our setup, where every agent reads about 25K shared visual tokens, KVCMAS stays the closest to NonShared, whereas recomputation methods lose substantial accuracy and KVComm runs out of memory (Table 5).

  • End-to-end latency and overhead (SabB, J1JH, pPqg): With full generation under load, KVCMAS achieves the lowest end-to-end latency and the highest throughput among the correction methods (Table 9), and it also has the lowest latency per generated token among them on the LLM benchmarks (Table 13). Its additional operations remain small compared with the dense prefill they replace (Table 16).

  • Dynamic workflows (pPqg): On replayed TraceLab sessions of coding-agent workloads, KVCMAS reduces the median TTFT by 1.9× over NonShared and achieves the lowest TTFT and end-to-end latency among all methods.

  • Quantization and offloading (pPqg): KV cache quantization is orthogonal and further reduces the memory of KVCMAS. Offloading prevents the OOM of KVComm but makes it slower than NonShared under load at long contexts, so the gains of KVCMAS do not come from avoiding OOM alone (Tables 22 and 23).

Results addressing the major concerns are included in the revised manuscript. We are grateful for the reviewers' engagement and hope this summary helps the final assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.