- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes AttuneKV, a receiver-side KV cache handoff method for multi-agent systems in which the sender and receiver share the same model. The method places the transferred message KV in the receiver's context with RoPE remapping, then forwards one existing receiver suffix token over the assembled cache. Its layer-wise attention ranks message positions for recomputation, while the attention-output change from removing each entry ranks positions for retention, so both decisions are obtained from a single token forward. The paper evaluates GSM8K, MMLU, and HumanEval across three multi-agent workflows, showing quality within 3.92 pp of dense prefill. On 12K-token messages, it reports a 2.92× receiver TTFT speedup over dense prefill and 1.45× over RelayCaching, with 2× lower handoff memory than dense prefill.
Critical strengths
- The method obtains its selection signal efficiently. Instead of processing all message rows through checking layers as in CacheBlend and RelayCaching, it uses one existing receiver token, which reduces the selection cost on the critical path. The controlled diagnostics are also well designed, comparing the one-token probe with the full suffix and an oracle on fixed events (Figure 3(a)), and showing that attention alone closely tracks an attention × true-deviation oracle (Figure 7).
Critical weaknesses
-
(Originality) The method is close to a combination of query-aware recomputation and value-aware eviction, and the distinction from these works could be clarified beyond the setting.
a. Selecting recomputation positions by the attention of a receiver-side query is closely related to ProphetKV and QCFuse, and the one-token probe resembles a SnapKV-style observation window of size one on a fixed template token.
b. The retention score follows the attention-output-error criterion used in CAOTE and VATP.
c. The mechanism does not appear to depend on the KV being generated by a sender rather than retrieved from a document, so it is not fully clear what is specific to the multi-agent handoff.
d. Eq. (3) is exact for removing a single entry under a fixed query. When 50% of entries are removed jointly and decoding uses queries different from , the rule becomes a heuristic, which could be stated more carefully.
-
(Quality / Significance) The quality and efficiency results are obtained in different regimes, and the memory comparison involves baselines that do not apply eviction.
a. Task quality is measured on short inter-agent messages, where all methods show similar TTFT (Figure 6(a)), while latency and memory are measured on 12K-token MATH windows without quality evaluation. It would be helpful to see both in the regime where the speedup appears.
b. The 2× memory saving comes from discarding half of the message KV, which could also be applied to the baselines. The Dense + Retention row suggests that this retention alone may reduce MMLU FC accuracy by 7.2 pp.
c. The 34× reduction over KVCOMM compares per-request state with a persistent pool that is designed to be amortized across requests.
d. MMLU appears to use 153 questions, where 3.92 pp corresponds to about six questions, and AttuneKV exceeding Dense in several cells suggests that some differences may be within variance.
Questions for authors
-
(ref Weakness 1) Which part of AttuneKV depends on the KV being sender-generated? How would it compare with ProphetKV- or QCFuse-style query-driven selection applied to the same handoff?
-
(ref Weakness 1) Does probing with a fixed end-of-turn token select different positions from probing with the receiver's task tokens? How stable is the retention decision as the decoding queries move away from over long outputs?
-
(ref Weakness 2) Could the authors report task quality on the 12K-token workload, and apply the same 50% retention to RelayCaching and CacheBlend for a comparison at the same retention level?
-
(ref Weakness 2) What is the MMLU sample size, and what is the seed-to-seed variance of the reported differences?
-
(Presentation, minor) Figure 2 seems to contain fragments of another figure (a "Resident cache state (GiB)" axis and bar panel) overlapping the diagram. The probing cost in Appendix A.2 is measured on an A100 while the main efficiency results use an H100, which could be clarified.
6 comments
Response to reviewer i37j (2/4)
W1b · CAOTE and VATP
Thank you for pointing this out. From CAOTE and VATP we learned that deciding whether a KV entry can be discarded requires considering the value in addition to attention. AttuneKV’s retention score directly adopts the formula derived in CAOTE, and the revised manuscript cites it next to Eqs. (3) and (5). VATP observed that attention sinks have high attention but small value norms, and therefore uses attention multiplied by the value norm as the importance score, while CAOTE goes further and derives the exact expression for the change in attention output after removing one token. Both works address KV cache eviction while the model is generated within a single context. We solve the different problem of KV handoff: a forward pass over just one token after the message decides recomputation and retention together, maintaining accuracy close to dense prefill while keeping only half of the message KV, with lower TTFT as well (Figure 6).
We added a score comparison at the same ρ = 50% (Table 6). The accuracy of removal, VATP-style, and attention-only are 82.4%, 82.4%, and 82.2%, with median ΔJ32 of 0.00017, 0.00021, and 0.00034, compared with 81.8% and 0.00261 for random. The three attention-based methods perform comparably, and AttuneKV adopts the removal criterion because it is the exact expression for the error of removing a single item.
Revisions to the paper: CAOTE is cited next to Eqs. (3) and (5), and Appendix A.4 adds Table 6.
Q2 · Single control token vs. task tokens, and stability of the retention decision
Thank you for these two very specific questions; we added a set of measurements for each. Using the receiver’s task tokens as the probe selects different positions, with only 45.3% overlap with the single-token selection, and it has lower coverage and takes longer to obtain. The single token’s coverage is 42.5% versus 27.7% for task tokens, and the time to obtain the probe is 21.2 versus 82.4 ms (table in the previous question). A fixed control token selects different positions in different handoffs, because its query is produced from the hidden state after reading the receiver’s prefix and the imported KV.
The retention decision is stable during decoding. Holding the imported KV fixed, we recompute the retention set with the query at each decoding step and compare it with the set at t₀ (averaged over the 512 events; Fig. R3, which is Figure 7 in Appendix A.5 of the revised manuscript). The overlap drops to about 88% at step 1 and then declines only slowly over the next 30 steps, reaching 81% at step 31 for the retention ranking alone and 82% for the actual retained set after adding the recomputed positions. Task outcomes also support this: in the long-message experiment, the decision agent answers 16 questions one by one, and adding retention does not reduce accuracy (Table 3).
Revisions to the paper: Appendix A.5 adds the stability of the retention set over decoding steps (Figure 7), and Appendix A.3 adds the comparison of probe sources.
Figures: Fig. R3: Stability of the retention set over decoding steps; at each step the retention set is recomputed with that step’s query and compared with the set chosen at handoff. link
W1d · Single-entry removal, joint eviction, and future queries
Thank you for the correction; the original manuscript was not precise enough on this point. The revised manuscript states next to Eq. (3) that it is exact for single-entry removal under a fixed query and is used as a ranking criterion for a joint removal of half entries. We have also added the exact expression for a joint removal set E:
The results of actually evicting half of the message KV jointly are listed in Table 6: the removal criterion has a median ΔJ32 of 0.00017, versus 0.00261 for random. The effect of query change is reported in Appendix A.5 (Figure 7).
Response to reviewer i37j (3/4)
Q3, W2a, W2b · Long-message quality and baselines at the same retention rate
Thank you for the suggestion. Measuring efficiency and accuracy separately in the original manuscript was a deliberate choice, and it is the same approach taken by KVCOMM and RelayCaching. In a multi-agent system, the messages each method reads are generated by upstream agents under that method, so their lengths differ, and TTFT depends not only on the algorithm but also on input length. Figure 6 therefore has all methods read the same input, and its 12K inputs are messages truncated to a fixed length, with no gradable answer. To answer the question of quality under long messages, we added a new long-message experiment in which the four agents of the original GSM8K-FC workflow process question sets composed of 16 problems each, and each method runs the full workflow on its own (83 question sets, 1,319 answers in total, on an H100). Under Dense, the decision agent’s input averages 10,911 tokens, with a maximum of 29,936 tokens. AttuneKV’s accuracy is 76.95% and Dense’s is 77.10% (Table 3).
Following your suggestion, we also added the same 50% retention to CacheBlend and RelayCaching on the same experiment. Both focus on recomputation and by design retain all of the message KV, so we tested two ways of retaining: keeping the same number of entries using AttuneKV’s removal criterion on the cache they have finished recomputing, and reusing their own KV-deviation signal to keep the half with the smallest KV change. Each method’s messages are generated by its own workflow, so the table also lists input tokens.
| Method | Accuracy | Input tokens (mean) | TTFT (ms) |
|---|---|---|---|
| RelayCaching | 76.35% | 8,674 | 122.74 |
| RelayCaching → AttuneKV retention step | 73.24% | 9,674 | 169.84 |
| RelayCaching + KV-deviation retention | 73.01% | 8,981 | 146.19 |
| CacheBlend | 75.21% | 7,817 | 92.44 |
| CacheBlend → AttuneKV retention step | 71.87% | 8,657 | 135.36 |
| CacheBlend + KV-deviation retention | 68.54% | 9,250 | 114.79 |
| AttuneKV | 76.95% | 9,715 | 112.92 |
At the same retention rate, the four retention variants reach accuracy of 68.54–73.24%, versus 76.95% for AttuneKV. Each method’s TTFT in the table corresponds to its own input length, so we measured the cost of the retention step at the 12,288-token point, where all methods read the same input (Figure 6(b) of the revised manuscript). CacheBlend and RelayCaching must run an extra probe after recomputation to obtain the retention scores. With AttuneKV’s retention step attached, they take 248 and 303 ms respectively, more than 60 ms above their original time. We run a forward pass over just one token after the message to obtain both the recomputation and retention signals, and with retention it takes 165 ms, 30 ms more than Recompute-only’s 135 ms.
Revisions to the paper: Table 3 in Section 4.5 reports the two retention variants, and Appendix B.7 gives their definitions and the attention breakdown figure.
Figures: Fig. R1: Revised Figure 6; panel (b) adds CacheBlend and RelayCaching with AttuneKV’s retention step. link Fig. R2: Breakdown of the decision agent’s later attention in the long-message experiment (Appendix B.7 of the revised manuscript). link
Response to reviewer i37j (1/4)
Thank you for your very thorough and careful review, and for recognizing the efficiency with which AttuneKV obtains its selection signal, as well as the controlled diagnostics in Figure 3(a) and Figure 7 (Figure 9 in the revised manuscript). Your questions about the role of sender-generated KV, the source of the probe, the comparison at the same retention rate, and statistical variance helped us define our contributions more clearly. We have added the corresponding experiments and, following your suggestions, corrected the statement of Eq. (3), Figure 2, and the hardware labeling.
Q1, W1a, W1c · Sender-generated KV, and ProphetKV, QCFuse, SnapKV
Thank you for this key question. In a handoff, the message’s KV is produced under the sender’s prefix and then read by a receiver with a different prefix. Our probe measures the receiver’s attention over this KV, and both recomputation and retention are decided from it. In RAG where query-driven methods (e.g., ProphetKV and QCFuse) are applied, chunk KV is computed independently for each chunk, so there is no prefix difference between a sender and a receiver. At handoff there is no user query after the message that could serve as the basis for selection, so we run a forward pass on only the first control token after the message. Beyond the setting, the methodological difference is that we use the same probe to decide recomputation and retention together before any recomputation occurs, whereas query-driven methods decide only recomputation and keep all of the KV. Besides, SnapKV focuses on the KV cache to be kept under a single, fixed prompt. We elaborate on each of the methods you mention below.
Revisions to the paper: The appendix adds the comparison with question tokens as the probe, and Section 5 spells out the differences of AttuneKV from ProphetKV, QCFuse, and SnapKV.