acceptodds
Under review as a conference paper at ICLR 2027

AttuneKV: Receiver-Aware KV-Cache Handoff for Efficient Multi-Agent Inference

Abstract

Multi-agent large language model (LLM) systems frequently process messages generated by other agents. When agents share a model, the sender’s key–value (KV) cache can be transferred to avoid redundant prefilling, but the imported cache is context-dependent and may not match the receiver’s context. Moreover, the receiver may need only part of the message during decoding. KV handoff therefore poses two receiver-side resource decisions: where to spend recomputation and which transferred entries to keep in memory. We present AttuneKV, a receiver-aware KV-cache handoff method that obtains both decisions from one lightweight receiver observation. AttuneKV forwards one existing receiver-side token over the imported cache. Its layerwise attention ranks message positions for recomputation, while the attention-output change induced by removing each entry ranks positions for retention. The same observation supports both decisions without forwarding the incoming message positions for selection or maintaining a persistent KV reference pool. At 12,288 incoming tokens, AttuneKV achieves a 2.92× receiver time to first token (TTFT) speedup over dense prefill and a 1.45× speedup over RelayCaching. It also reduces KV-handoff memory by 2.0× and 34× relative to dense prefill and KVCOMM, respectively. Across math problem solving, broad-domain knowledge evaluation, and code generation, AttuneKV remains within 3.92 pp of dense prefill performance.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Multi-Agent Systems, KV Cache Reuse, Efficient Inference

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_i37jOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes AttuneKV, a receiver-side KV cache handoff method for multi-agent systems in which the sender and receiver share the same model. The method places the transferred message KV in the receiver's context with RoPE remapping, then forwards one existing receiver suffix token over the assembled cache. Its layer-wise attention ranks message positions for recomputation, while the attention-output change from removing each entry ranks positions for retention, so both decisions are obtained from a single token forward. The paper evaluates GSM8K, MMLU, and HumanEval across three multi-agent workflows, showing quality within 3.92 pp of dense prefill. On 12K-token messages, it reports a 2.92× receiver TTFT speedup over dense prefill and 1.45× over RelayCaching, with 2× lower handoff memory than dense prefill.

Critical strengths
  1. The method obtains its selection signal efficiently. Instead of processing all message rows through checking layers as in CacheBlend and RelayCaching, it uses one existing receiver token, which reduces the selection cost on the critical path. The controlled diagnostics are also well designed, comparing the one-token probe with the full suffix and an oracle on fixed events (Figure 3(a)), and showing that attention alone closely tracks an attention × true-deviation oracle (Figure 7).
Critical weaknesses
  1. (Originality) The method is close to a combination of query-aware recomputation and value-aware eviction, and the distinction from these works could be clarified beyond the setting.

    a. Selecting recomputation positions by the attention of a receiver-side query is closely related to ProphetKV and QCFuse, and the one-token probe resembles a SnapKV-style observation window of size one on a fixed template token.

    b. The retention score follows the attention-output-error criterion used in CAOTE and VATP.

    c. The mechanism does not appear to depend on the KV being generated by a sender rather than retrieved from a document, so it is not fully clear what is specific to the multi-agent handoff.

    d. Eq. (3) is exact for removing a single entry under a fixed query. When 50% of entries are removed jointly and decoding uses queries different from t0t_0, the rule becomes a heuristic, which could be stated more carefully.

  2. (Quality / Significance) The quality and efficiency results are obtained in different regimes, and the memory comparison involves baselines that do not apply eviction.

    a. Task quality is measured on short inter-agent messages, where all methods show similar TTFT (Figure 6(a)), while latency and memory are measured on 12K-token MATH windows without quality evaluation. It would be helpful to see both in the regime where the speedup appears.

    b. The 2× memory saving comes from discarding half of the message KV, which could also be applied to the baselines. The Dense + Retention row suggests that this retention alone may reduce MMLU FC accuracy by 7.2 pp.

    c. The 34× reduction over KVCOMM compares per-request state with a persistent pool that is designed to be amortized across requests.

    d. MMLU appears to use 153 questions, where 3.92 pp corresponds to about six questions, and AttuneKV exceeding Dense in several cells suggests that some differences may be within variance.

Questions for authors
  1. (ref Weakness 1) Which part of AttuneKV depends on the KV being sender-generated? How would it compare with ProphetKV- or QCFuse-style query-driven selection applied to the same handoff?

  2. (ref Weakness 1) Does probing with a fixed end-of-turn token select different positions from probing with the receiver's task tokens? How stable is the retention decision as the decoding queries move away from t0t_0 over long outputs?

  3. (ref Weakness 2) Could the authors report task quality on the 12K-token workload, and apply the same 50% retention to RelayCaching and CacheBlend for a comparison at the same retention level?

  4. (ref Weakness 2) What is the MMLU sample size, and what is the seed-to-seed variance of the reported differences?

  5. (Presentation, minor) Figure 2 seems to contain fragments of another figure (a "Resident cache state (GiB)" axis and bar panel) overlapping the diagram. The probing cost in Appendix A.2 is measured on an A100 while the main efficiency results use an H100, which could be clarified.

6 comments
AuthorsOpenReview ↗

Response to reviewer i37j (1/4)

Thank you for your very thorough and careful review, and for recognizing the efficiency with which AttuneKV obtains its selection signal, as well as the controlled diagnostics in Figure 3(a) and Figure 7 (Figure 9 in the revised manuscript). Your questions about the role of sender-generated KV, the source of the probe, the comparison at the same retention rate, and statistical variance helped us define our contributions more clearly. We have added the corresponding experiments and, following your suggestions, corrected the statement of Eq. (3), Figure 2, and the hardware labeling.

Q1, W1a, W1c · Sender-generated KV, and ProphetKV, QCFuse, SnapKV

Thank you for this key question. In a handoff, the message’s KV is produced under the sender’s prefix and then read by a receiver with a different prefix. Our probe measures the receiver’s attention over this KV, and both recomputation and retention are decided from it. In RAG where query-driven methods (e.g., ProphetKV and QCFuse) are applied, chunk KV is computed independently for each chunk, so there is no prefix difference between a sender and a receiver. At handoff there is no user query after the message that could serve as the basis for selection, so we run a forward pass on only the first control token after the message. Beyond the setting, the methodological difference is that we use the same probe to decide recomputation and retention together before any recomputation occurs, whereas query-driven methods decide only recomputation and keep all of the KV. Besides, SnapKV focuses on the KV cache to be kept under a single, fixed prompt. We elaborate on each of the methods you mention below.

  • ProphetKV. ProphetKV starts from the observation that “cross-attention utility is inherently query-contingent”: the user query following the documents determines which document tokens are worth recomputing, and the same document should have different positions recomputed for different questions, so it uses the user query’s tokens for selection. This observation was very inspiring to us, and it also shows exactly that the reasons for recomputation differ between ProphetKV and AttuneKV. ProphetKV recomputes because each chunk’s KV is computed in advance independently, and a question that appears later changes which content matters. In a multi-agent handoff there is no such question after the message; what the receiver will do is already written in its own prefix. AttuneKV recomputes because the message’s KV was produced under the sender’s prefix while the receiver has a different prefix, and what needs to be corrected is the difference between these two prefixes. Therefore, AttuneKV’s probe carries the receiver’s prefix to read the sender-generated KV. We also tested a ProphetKV-style approach on the same 512 events, namely appending the question tokens after the message and running one more forward pass as the probe. The results are in the table below: the positions it selects capture less subsequent attention, and obtaining the probe takes longer. In addition, ProphetKV keeps all KV, whereas AttuneKV goes on to decide retention from the same observation.
  • QCFuse. QCFuse likewise centers on the query to select positions for recomputation and achieves strong results on RAG. It pre-builds anchors for each reusable chunk and calibrates the key layers offline, which suits cases where chunks are documents known in advance. Handoff messages, by contrast, are generated online, so anchors would need to be built for each new message at every handoff, whereas AttuneKV reads the imported cache directly and can save the anchors and the offline calibration.
  • SnapKV addresses which parts of the KV cache to keep for a single, fixed prompt, whereas AttuneKV addresses handoff between agents, where the sender and receiver prefixes differ. In SnapKV’s setting the prompt is fixed, and the task is to select which parts of its KV cache can be evicted (at,i=Attn(q(t∣c), ki(m∣c))a_{t,i} = \mathrm{Attn}(q(t \mid c),\, k_i(m \mid c))). In the handoff setting, the cache is produced under the sender’s prefix and then read by a receiver with a different prefix (ai=Attn(q(t0∣cR), ki(m∣cS))a_i = \mathrm{Attn}(q(t_0 \mid c_R),\, k_i(m \mid c_S))). Prior work has mainly addressed how to recompute this cache, and AttuneKV goes further by considering how to decide recomputation and retention together.
ProbeLengthCoverageOverlap with single-token selectionTime to obtain probe
Single token (AttuneKV)1 token42.5%—21.2 ms
Full suffixMedian 5 tokens42.6%92.4%31.6 ms
Question tokens appended after the messageMedian 54.5 tokens27.7%45.3%82.4 ms

Revisions to the paper: The appendix adds the comparison with question tokens as the probe, and Section 5 spells out the differences of AttuneKV from ProphetKV, QCFuse, and SnapKV.

AuthorsOpenReview ↗

Response to reviewer i37j (2/4)

W1b · CAOTE and VATP

Thank you for pointing this out. From CAOTE and VATP we learned that deciding whether a KV entry can be discarded requires considering the value in addition to attention. AttuneKV’s retention score directly adopts the formula derived in CAOTE, and the revised manuscript cites it next to Eqs. (3) and (5). VATP observed that attention sinks have high attention but small value norms, and therefore uses attention multiplied by the value norm as the importance score, while CAOTE goes further and derives the exact expression for the change in attention output after removing one token. Both works address KV cache eviction while the model is generated within a single context. We solve the different problem of KV handoff: a forward pass over just one token after the message decides recomputation and retention together, maintaining accuracy close to dense prefill while keeping only half of the message KV, with lower TTFT as well (Figure 6).

We added a score comparison at the same ρ = 50% (Table 6). The accuracy of removal, VATP-style, and attention-only are 82.4%, 82.4%, and 82.2%, with median ΔJ32 of 0.00017, 0.00021, and 0.00034, compared with 81.8% and 0.00261 for random. The three attention-based methods perform comparably, and AttuneKV adopts the removal criterion because it is the exact expression for the error of removing a single item.

Revisions to the paper: CAOTE is cited next to Eqs. (3) and (5), and Appendix A.4 adds Table 6.

Q2 · Single control token vs. task tokens, and stability of the retention decision

Thank you for these two very specific questions; we added a set of measurements for each. Using the receiver’s task tokens as the probe selects different positions, with only 45.3% overlap with the single-token selection, and it has lower coverage and takes longer to obtain. The single token’s coverage is 42.5% versus 27.7% for task tokens, and the time to obtain the probe is 21.2 versus 82.4 ms (table in the previous question). A fixed control token selects different positions in different handoffs, because its query is produced from the hidden state after reading the receiver’s prefix and the imported KV.

The retention decision is stable during decoding. Holding the imported KV fixed, we recompute the retention set with the query at each decoding step and compare it with the set at t₀ (averaged over the 512 events; Fig. R3, which is Figure 7 in Appendix A.5 of the revised manuscript). The overlap drops to about 88% at step 1 and then declines only slowly over the next 30 steps, reaching 81% at step 31 for the retention ranking alone and 82% for the actual retained set after adding the recomputed positions. Task outcomes also support this: in the long-message experiment, the decision agent answers 16 questions one by one, and adding retention does not reduce accuracy (Table 3).

Revisions to the paper: Appendix A.5 adds the stability of the retention set over decoding steps (Figure 7), and Appendix A.3 adds the comparison of probe sources.

Figures: Fig. R3: Stability of the retention set over decoding steps; at each step the retention set is recomputed with that step’s query and compared with the set chosen at handoff. link

W1d · Single-entry removal, joint eviction, and future queries

Thank you for the correction; the original manuscript was not precise enough on this point. The revised manuscript states next to Eq. (3) that it is exact for single-entry removal under a fixed query and is used as a ranking criterion for a joint removal of half entries. We have also added the exact expression for a joint removal set E:

o−E−o=∑i∈Eai(o−vi)1−∑i∈Eaio_{-E} - o = \frac{\sum_{i \in E} a_i (o - v_i)}{1 - \sum_{i \in E} a_i}

The results of actually evicting half of the message KV jointly are listed in Table 6: the removal criterion has a median ΔJ32 of 0.00017, versus 0.00261 for random. The effect of query change is reported in Appendix A.5 (Figure 7).

AuthorsOpenReview ↗

Response to reviewer i37j (3/4)

Q3, W2a, W2b · Long-message quality and baselines at the same retention rate

Thank you for the suggestion. Measuring efficiency and accuracy separately in the original manuscript was a deliberate choice, and it is the same approach taken by KVCOMM and RelayCaching. In a multi-agent system, the messages each method reads are generated by upstream agents under that method, so their lengths differ, and TTFT depends not only on the algorithm but also on input length. Figure 6 therefore has all methods read the same input, and its 12K inputs are messages truncated to a fixed length, with no gradable answer. To answer the question of quality under long messages, we added a new long-message experiment in which the four agents of the original GSM8K-FC workflow process question sets composed of 16 problems each, and each method runs the full workflow on its own (83 question sets, 1,319 answers in total, on an H100). Under Dense, the decision agent’s input averages 10,911 tokens, with a maximum of 29,936 tokens. AttuneKV’s accuracy is 76.95% and Dense’s is 77.10% (Table 3).

Following your suggestion, we also added the same 50% retention to CacheBlend and RelayCaching on the same experiment. Both focus on recomputation and by design retain all of the message KV, so we tested two ways of retaining: keeping the same number of entries using AttuneKV’s removal criterion on the cache they have finished recomputing, and reusing their own KV-deviation signal to keep the half with the smallest KV change. Each method’s messages are generated by its own workflow, so the table also lists input tokens.

MethodAccuracyInput tokens (mean)TTFT (ms)
RelayCaching76.35%8,674122.74
RelayCaching → AttuneKV retention step73.24%9,674169.84
RelayCaching + KV-deviation retention73.01%8,981146.19
CacheBlend75.21%7,81792.44
CacheBlend → AttuneKV retention step71.87%8,657135.36
CacheBlend + KV-deviation retention68.54%9,250114.79
AttuneKV76.95%9,715112.92

At the same retention rate, the four retention variants reach accuracy of 68.54–73.24%, versus 76.95% for AttuneKV. Each method’s TTFT in the table corresponds to its own input length, so we measured the cost of the retention step at the 12,288-token point, where all methods read the same input (Figure 6(b) of the revised manuscript). CacheBlend and RelayCaching must run an extra probe after recomputation to obtain the retention scores. With AttuneKV’s retention step attached, they take 248 and 303 ms respectively, more than 60 ms above their original time. We run a forward pass over just one token after the message to obtain both the recomputation and retention signals, and with retention it takes 165 ms, 30 ms more than Recompute-only’s 135 ms.

Revisions to the paper: Table 3 in Section 4.5 reports the two retention variants, and Appendix B.7 gives their definitions and the attention breakdown figure.

Figures: Fig. R1: Revised Figure 6; panel (b) adds CacheBlend and RelayCaching with AttuneKV’s retention step. link Fig. R2: Breakdown of the decision agent’s later attention in the long-message experiment (Appendix B.7 of the revised manuscript). link

Reviewer_qFpnOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies KV-cache handoff between agents that share one model. AttuneKV forwards one existing receiver-suffix token over the imported cache, uses its attention to pick 10% of message positions to recompute, and keeps 50% of positions ranked by a removal score. On an H100 with Llama-3.1-8B and 12,288 incoming MATH-trace tokens, receiver TTFT is 165.9 ms vs. 484.3 ms for dense prefill (2.92x) and 240.6 ms for RelayCaching (1.45x), and message KV drops from 1.50 to 0.75 GiB.

Critical strengths
  1. The recomputation signal is cheap and well supported. It costs one query row per layer instead of a pass over all message rows, which matters for long messages.
  2. The systems accounting is very careful. Six baselines share one framework, TTFT includes every phase, and the paper reports that Recompute-only is faster than full AttuneKV (which makes sense to me).
Critical weaknesses
  1. Speed and quality are never measured at the same operating point. All TTFT and memory numbers come from 5 fixed MATH-500 tasks with up to 12,288 message tokens, but quality on these inputs is not reported. Quality comes from GSM8K/MMLU/HumanEval with unstated message lengths, and at 512 tokens all methods are below about 50 ms.

  2. The retention half reuses an existing score and is not compared to alternatives. Eqs. 3 and 5 match the closed-form CAOTE score, but Sec. 5 cites CAOTE only in general terms. Since CacheBlend, EPIC and RelayCaching keep all message KV, the 2.0x memory gain comes from the ρ knob, and retention costs about 30 ms, more than the whole margin over CacheBlend.

Questions for authors
  1. What is the final-agent accuracy on the same 512-12,288-token inputs used for Fig. 6, and what are the mean and maximum message lengths in the Table 1 runs?
  2. What are TTFT and quality for CacheBlend and RelayCaching with the same ρ = 50% eviction applied?
  3. Were the 512 GSM8K diagnostic questions drawn from the test set used in Table 1?
3 comments
AuthorsOpenReview ↗

Response to reviewer qFpn (1/3)

Thank you for your review, and for recognizing the low cost of the recomputation signal and our careful handling of system timing. Your comments on quality under long messages, baselines at the same retention rate, and the positioning of the retention step were all specific, and we respond to each with a new set of experiments.

Q1, W1 · Long-message quality and native message lengths

Thank you for the question. In the original manuscript, we chose to report efficiency and accuracy in separate experiments, following the practice of KVCOMM and RelayCaching. The reason is that in a multi-agent pipeline, each method receives messages written by upstream agents running under that same method, so the inputs differ in length from one method to another. Note that TTFT is influenced by the algorithm itself as well as by input length. Comparing the efficiency of the methods themselves requires all methods to read the same input. This is what Figure 6 does, and its 512–12,288-token inputs are messages truncated to fixed lengths, with no gradable answer. Since Table 1 mainly reports accuracy on short messages, we added a new long-message experiment to answer the question of accuracy under long messages. The four agents of the original GSM8K-FC workflow process question sets composed of 16 problems each, and each method runs the full workflow on its own (83 question sets, 1,319 answers in total, on an H100). Under Dense, the decision agent’s input averages 10,911 tokens, with a maximum of 29,936 tokens. The table below places each method’s accuracy, input tokens, and TTFT side by side.

MethodAccuracyInput tokens (mean)TTFT (ms)
Dense77.10%10,911432.87
KVCOMM76.27%9,055185.67
RelayCaching76.35%8,674122.74
CacheBlend75.21%7,81792.44
EPIC55.57%6,75518.18
Direct Reuse38.82%8,03918.05
AttuneKV76.95%9,715112.92

On this long-message workload, AttuneKV’s accuracy is 76.95%, just 0.15 pp below Dense’s 77.10%, while retaining only half of the message KV (Table 3). Each method’s TTFT in the table corresponds to its own input length, and speed comparisons between methods rest on Figure 6, where all methods read the same input.

Across the nine settings of the original Table 1, the decision agent’s mean input length is 825–3,725 tokens, and the maximum is 2,225–24,808 tokens (Dense, including the question and upstream messages). At these lengths, Dense’s TTFT is 38.5–130.8 ms and AttuneKV’s is 36.6–45.1 ms. In seven of the settings the mean input is below 2,000 tokens, and every method’s TTFT is in the tens of milliseconds. Efficiency differences only open up at longer inputs, which is why we added the long-message experiment.

Revisions to the paper: Appendix B.6 adds the input length and TTFT of each workflow of the original Table 1 (Table 9). Section 4 adds a Long-Message Handoff subsection (4.5) with Table 3.

AuthorsOpenReview ↗

Response to reviewer qFpn (2/3)

Q2, W2 · CacheBlend and RelayCaching at the same ρ = 50%

Following your suggestion, we added the same 50% retention to CacheBlend and RelayCaching on the long-message experiment, and measured TTFT and accuracy in the same run. Both focus on recomputation and by design retain all of the message KV, so we tested two ways of retaining. In the first, we run one more forward pass of AttuneKV’s probe on the cache they have finished recomputing, then keep the same number of message KV entries as AttuneKV using the removal criterion. In the second, we reuse their own KV-deviation signal and keep the half with the smallest KV change. Below please find a summary table. The table includes input tokens, since each method’s messages are generated by its own workflow.

MethodAccuracyInput tokens (mean)TTFT (ms)
RelayCaching76.35%8,674122.74
RelayCaching → AttuneKV retention step73.24%9,674169.84
RelayCaching + KV-deviation retention73.01%8,981146.19
CacheBlend75.21%7,81792.44
CacheBlend → AttuneKV retention step71.87%8,657135.36
CacheBlend + KV-deviation retention68.54%9,250114.79
Recompute-only76.35%9,28793.13
AttuneKV76.95%9,715112.92

To answer your question: with 50% retention added, the four retention variants of the two baselines reach 68.54–73.24% accuracy, compared with 76.95% for AttuneKV. On timing, each method’s TTFT in the table corresponds to its own input length, so we measured the cost of the retention step at the 12,288-token point, where all methods read the same input (Figure 6(b)). CacheBlend and RelayCaching must run an extra probe after recomputation to obtain the retention scores. With AttuneKV’s retention step attached, they take 248 and 303 ms respectively, more than 60 ms above their original time. AttuneKV’s recomputation ranking and retention ranking come from the same probe, and with retention it takes 165 ms, 30 ms more than Recompute-only’s 135 ms.

The receiver’s subsequent attention shows why the two baselines need this extra probe. We divided the decision agent’s attention to message tokens during decoding into three categories: attention falling on recomputed KV, on retained but not recomputed KV, and on discarded KV. Attention is defined the same way as in Figure 3(a).

MethodRecomputedRetained but not recomputedDiscarded
CacheBlend22%78%—
RelayCaching45%55%—
Recompute-only57%43%—
CacheBlend → AttuneKV retention step24%64%12%
RelayCaching → AttuneKV retention step45%43%12%
CacheBlend + KV-deviation retention22%34%44%
RelayCaching + KV-deviation retention44%20%36%
AttuneKV57%31%12%

AttuneKV covers 57% of subsequent attention with a 10% recomputation budget, whereas CacheBlend covers 22% with the same budget, and RelayCaching recomputes about 16% of positions and covers 45%. By running a forward pass over just one token after the message, AttuneKV obtains both signals from the same attention: which positions are worth recomputing and which can be discarded. The recomputed positions therefore cover the 57% of subsequent attention noted above, the discarded positions receive only 12%, and the remaining 31% falls on retained but not recomputed KV. CacheBlend and RelayCaching are designed for reuse, and their main signal is KV deviation, which reflects how much the KV changed between the two contexts. When it is used to decide retention, the discarded positions receive 36–44% of subsequent attention. To obtain the same retention signal as AttuneKV, CacheBlend and RelayCaching require an additional probe after recomputation (Figure 6(b)).

Revisions to the paper: Table 3 in Section 4.5 reports the two retention variants, and Appendix B.7 gives their definitions and the attention breakdown figure.

Figures: Fig. R1: Revised Figure 6; panel (b) adds CacheBlend and RelayCaching with AttuneKV’s retention step. link Fig. R2: Breakdown of the decision agent’s later attention in the long-message experiment (Appendix B.7 of the revised manuscript). link

AuthorsOpenReview ↗

Response to reviewer qFpn (3/3)

W2 · The CAOTE criterion and the contribution of the retention step

Thank you for the reminder. The retention criterion directly adopts the formula derived in CAOTE, and the original manuscript did not cite it explicitly enough; the revised manuscript now cites it directly next to Eqs. (3) and (5). AttuneKV applies this criterion to KV handoff: a forward pass over just one token after the message decides recomputation and retention together, maintaining accuracy close to dense prefill while keeping only half of the message KV.

We have also added a comparison of retention scores (Table 6). With the recomputation result and the number of retained entries fixed, the accuracy of the removal, VATP-style, and attention-only is 82.4%, 82.4%, and 82.2%, with median ΔJ32 of 0.00017, 0.00021, and 0.00034, compared with 81.8% and 0.00261 for random. The differences among the retention scores are small, which is consistent with the small gains observed by the original CAOTE paper on top of SnapKV.

Revisions to the paper: CAOTE is cited next to Eqs. (3) and (5), Section 5 explains the differences between the two, and Appendix A.4 adds Table 6.

W2 · The roughly 30 ms retention cost and the margin over CacheBlend

Thank you for carefully checking these two numbers. The purpose of the retention step is to halve the resident message KV. Its cost is already included in the TTFT we report, and AttuneKV with retention is still faster than CacheBlend keeping all KV. As you point out, the retention step takes about 30 ms at the 12K point, which is the price paid for halving the message KV.

CacheBlend and RelayCaching focus on recomputation and by design retain all of the message KV; they do not discuss discarding KV cache to save memory. If retention is attached after recomputation so that the three methods are compared at the same retention rate, CacheBlend and RelayCaching need an additional probe after recomputation to obtain the retention scores, which costs more than 60 ms extra on the same 12,288-token input. AttuneKV’s recomputation ranking and retention ranking come from the same probe, so retention costs only 30 ms more than Recompute-only (Figure 6(b)). The time and accuracy of each method are listed in the table below.

MethodReceiver time with retention (ms, 12,288 tokens, Figure 6(b))Accuracy, keeping allAccuracy, keeping 50%
AttuneKV16576.35%76.95%
RelayCaching30376.35%73.24%
CacheBlend24875.21%71.87%

Revisions to the paper: Section 4.3 states the cost of the retention step, and Section 4.5 adds the same-retention-rate comparison (Table 3).

Figures: Fig. R1: Revised Figure 6; panel (b) adds CacheBlend and RelayCaching with AttuneKV’s retention step. link

Q3 · Whether the diagnostics overlap with Table 1

The 512 diagnostic events come from the GSM8K training split, and Table 1 uses the test split, so the two do not overlap. The original manuscript did not state this. Thank you for the reminder; the revised manuscript adds it to Section 2.

Reviewer_g5uyOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper studies how to make the KV-cache-based communication of multi-agent systems more efficient by selectively recomputing and discarding some of the KV cache entries in the message handoff. The idea is that they first do a single-token forward pass on the receiver that attends to all the tokens in the message and find out which positions are worth recomputing and retaining. The selection of recompute positions is based on the attention score of the positions, where their preliminary experiments show that the attention score of the tokens selected by single-token pass is at least better than random selection. The selection of retained positions is based on the JS divergence of the output after removing the token. Experiments show that the proposed method reduces TTFS and memory footprints compared to the baselines with some performance loss.

Critical strengths
  • The single-token probing method for selecting the positions to recompute and retain is well motivated in the preliminary section.
  • The experimental results are solid, showing notable TTFT and memory reductions compared to the baselines.
Critical weaknesses
  • The method relies on the heuristic of the attention score of a single token. While Figure 3 showed that the single token performs similarly to the full suffix baseline, it still has a notable gap of 12 points gap compared to the oracle setting, suggesting that certain tokens become more important later in the generation.
  • The related-work discussion could better position the proposed method with respect to prior single-token probing approaches. Recent methods such as ReToken[1] and KV-PRM[2] similarly use a single token to probe a much larger cached context and derive useful signals for their tasks.

[1] ReToken: One Token to Improve Vision–Language Models for Visual Retrieval, NeurIPS 2026

[2] KV-PRM: Efficient Process Reward Modeling via KV-Cache Transfer for Multi-Agent Test-Time Scaling, NeurIPS 2026

Questions for authors
  • Can the author analyze where the 12-point gap between the single-token probe and oracle baseline is coming from? So that we know how sufficient a single-token probe is for identifying the important tokens during generation.
  • Can the authors discuss the proposed single-token probing method with prior work for better technical contribution positioning?
3 comments
AuthorsOpenReview ↗

Response to reviewer g5uy

Thank you for your positive assessment, and for recognizing that the single-token probe is well motivated in the preliminary study and that the experimental results are solid. Your observation about the gap between the single token and the Oracle is insightful. Below we explain the two sources of this gap, and we have added a discussion to Related Work about the positioning of the proposed single-token probing method with respect to ReToken and KV-PRM.

Q1, W1 · The 12-point gap between the single-token probe and the Oracle

Thank you for this insightful question. The 12-point gap arises because the Oracle and the single-token probe have access to different information at handoff time, and the difference has two sources.

First, the Oracle reads the KV cache after dense prefill. The Oracle can first recompute a complete KV cache under the receiver’s context (the ground truth) to determine the attention weights. In practice, every KV cache reuse algorithm exists to avoid this dense prefill, so what can be read at handoff time is only the KV cache influenced by the sender’s prefix. The absence of this ground-truth cache is the main cause of the performance difference.

Second, the Oracle uses the queries of the following 32 decoding tokens. The Oracle’s ranking is taken directly from the attention that actually occurs during the receiver’s next 32 decoding steps. As you observed, some positions only become important after generation has progressed. At handoff time, only a small control-token suffix follows the message (five tokens on the Llama-3.1-8B-Instruct we use), and obtaining queries for more tokens would require a start of decoding.

Within the information available at handoff time, a single token is already sufficient. The full suffix probe uses all queries available at that moment, yet its coverage is only 0.1 pp higher than that of the single token (as shown in table below). As for the Oracle, the 12 points come from the two pieces of information above, which do not exist at handoff time.

Selection methodQueries usedMessage KV readCoverage
Single-token probeFirst suffix token after the messageKV influenced by the sender prefix42.5%
Full suffix probeEntire suffixKV influenced by the sender prefix42.6%
OracleQueries from the next 32 decoding stepsKV from dense prefill54.9%

Revisions to the paper: Section 2.1 adds the definition of the Oracle, and Appendix A.3 explains how these two information differences account for the 12-point gap.

Q2, W2 · Positioning relative to ReToken and KV-PRM

Thank you for pointing out ReToken and KV-PRM. Both also use a single token to read an existing KV cache and are closely related to our probe, and it was our oversight not to discuss them in the original manuscript. The key distinction is that ReToken [1] and KV-PRM [2] train a readout mechanism to extract a task answer from a cache that is correct for the reader, whereas AttuneKV uses an existing, untrained token to measure how the receiver will read a cache produced under a different context, and decides from this which positions to recompute and retain.

ReToken trains a retrieval token and a projection matrix to select the frames to load when answering, and KV-PRM trains a LoRA adapter so that a verify token outputs a process reward. In both, the cache being read is the reader’s own input prefix, so neither deals with context mismatch. In AttuneKV, the cache comes from the sender’s prefix, and the probe must identify at which positions this deviation affects the receiver’s output. This quantity can be computed directly from attention, because the first-order effect of recomputation has aia_i as its coefficient (Eq. (2)), and the effect of removal is exactly rir_i (Eq. (3)). A single forward pass at t0t_0 therefore yields both rankings for recomputation and retention without any training, and widening the probe to the entire suffix barely changes the results (Figure 3(a), Appendix A.2). KV-PRM’s KV Steering also modifies the cache at handoff, but it uses gradients to update the whole cache to raise the scorer’s score, whereas AttuneKV recomputes only the positions selected by the probe, bringing the receiver’s prediction close to that of dense prefill.

Revisions to the paper: Section 5 adds a new paragraph with this comparison and cites the two works. [1] ReToken: One Token to Improve Vision–Language Models for Visual Retrieval, NeurIPS 2026. [2] KV-PRM: Efficient Process Reward Modeling via KV-Cache Transfer for Multi-Agent Test-Time Scaling, NeurIPS 2026.

Thank you for the response. My concerns are mostly addressed. I will maintain my positive rating.

AuthorsOpenReview ↗

Thank you for your feedback

Thank you for reading our response and for your positive assessment. Your questions on the gap to the Oracle and on ReToken and KV-PRM helped us sharpen the paper, and both are now reflected in the revised manuscript. If any point remains unclear, we would be glad to clarify it during the discussion period.

Reviewer_zhmdOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

AttuneKV handles KV handoff between agents that share a model. It runs one existing receiver token over the imported, RoPE-remapped cache. Its attention picks 10% of message positions to recompute, and a removal-sensitivity score decides which 50% of entries to keep. On Llama-3.1-8B and Qwen2.5-Coder-7B (GSM8K, MMLU, HumanEval, three workflows), quality stays close to dense prefill. At 12,288 tokens the paper reports 2.92× TTFT over dense and 1.45× over RelayCaching, with half the message KV memory.

Critical strengths

The idea is a clean and practical framing, which treats recompute and retention as two separate budgets, both decided by one cheap probe. The one-token probe costs about the same as a full-suffix probe (32.7 vs 35.3 ms). Timing is accounted for carefully, and the controls are useful.

Critical weaknesses
  1. Novelty is limited and not stated clearly. The recompute score comes from KVShare. The retention score a/(1−a)·‖v−o‖ is the same form as CAOTE's eviction score. Selecting by the last token's attention is essentially SnapKV with a window of one. The retention score is also never compared with attention-only, VATP or random retention at the same ρ, so the second contribution is not really tested.

  2. Speed and quality come from different workloads. The speedups use five MATH tasks replayed at 12k tokens, with no quality measured. Quality is measured on few-hundred-token messages, with no TTFT. The 2× memory saving follows directly from ρ=0.5 while the baselines keep everything, and the 34× over KVCOMM depends on its anchor setting and an amortized pool. The abstract compares against RelayCaching (1.45×), although CacheBlend is closer (1.12×). "Within 3.92 pp" refers to a cell where AttuneKV beats dense; the largest actual drop is 3.56 pp.

Questions for authors
  1. What is task accuracy on the 12k-token workload used for timing?
  2. What is TTFT on the Table 1 workflows at their real message lengths?
  3. At the same ρ, how does r_i compare with attention-only retention?
  4. What is KVCOMM's anchor hit rate in the efficiency setting?
  5. How large is the MMLU subset, and which split do the 512 diagnostic GSM8K questions come from?
5 comments
AuthorsOpenReview ↗

Response to reviewer zhmd (1/3)

Thank you for your review, and for recognizing the idea behind AttuneKV: treating recomputation and retention as two separate budgets, both determined by a single low-cost probe, along with our careful handling of timing. Your comments on how we position the novelty, how we measure speed and quality, and the fairness of the memory comparison were all constructive, and responding to them has made the paper’s claims clearer. We have added new experiments addressing these points and corrected two numbers in the abstract as you pointed out. Our item-by-item responses are below.

W1, Q3 · Novelty and comparison of retention scores

Thank you for carefully comparing our work against KVShare, SnapKV, and CAOTE. The original manuscript did not make clear how AttuneKV differs from these three works, and that was our oversight. Each of these works proposes an elegant solution to its own problem, whereas AttuneKV addresses a different one: handing off KV computed by a sender to a receiver whose prefix is different. Below, we first state in one sentence what AttuneKV adds, then address each of the three works you mention, and finally answer Q3.

AttuneKV’s new contribution is to use a single observation of the receiver at handoff time to decide recomputation and retention together. What matters at handoff is which positions of the message the receiver will read next, and the answer to this one question determines both recomputation and retention. AttuneKV observes it by running one forward pass on the first suffix token after the message, so both decisions are available before any recomputation takes place. Even after discarding half of the message KV on this basis, task accuracy remains comparable to dense prefill. The three similarities you point out are addressed one by one below.

  • KVShare’s score requires the value deviation, whereas AttuneKV needs only attention. KVShare uses si=ai⋅∣ΔVi∣1s_i = a_i \cdot |\Delta V_i|_1 as the criterion for selecting tokens to recompute. However, an exact ΔVi\Delta V_i requires the Value of the token recomputed under the receiver’s context, to be compared with the reused Value. AttuneKV instead uses aia_i directly and completes the ranking with a forward pass over a single token. Figure 7 of the original manuscript (Figure 9 in the revised manuscript) shows that the attention-only result is very close to the oracle that uses the true deviation.
  • SnapKV addresses which parts of the KV cache to keep for a single, fixed prompt, whereas AttuneKV addresses handoff between agents, where the sender and receiver prefixes differ. In SnapKV’s setting the prompt is fixed, and the task is to select which parts of its KV cache can be evicted (at,i=Attn(q(t∣c), ki(m∣c))a_{t,i} = \mathrm{Attn}(q(t \mid c),\, k_i(m \mid c))). Note that in the handoff setting, the cache is produced under the sender’s prefix and then read by a receiver with a different prefix (ai=Attn(q(t0∣cR), ki(m∣cS))a_i = \mathrm{Attn}(q(t_0 \mid c_R),\, k_i(m \mid c_S))). Prior work has mainly addressed how to recompute this cache, while AttuneKV goes further by considering how to decide recomputation and retention together.
  • Thank you for pointing out that AttuneKV’s retention score directly adopts the formula derived in CAOTE. Our ri=ai1−ai∥vi−o∥r_i = \frac{a_i}{1-a_i} \| v_i - o \| comes from it, and the revised manuscript cites it explicitly next to Eqs. (3) and (5). What we solve is the different problem of KV handoff: a forward pass over just one token after the message decides recomputation and retention together, maintaining accuracy while keeping only half of the message KV, with lower TTFT as well (Figure 6).

Answer to Q3: at the same ρ = 50%, r_i performs comparably to attention-only retention, and both have a clearly lower ΔJ32 than random. Thank you for pointing out that the original manuscript lacked this comparison. We added experiments for the three baselines you listed. We hold fixed the probe, the recomputation set, the post-recomputation cache, and the number of retained entries, and vary only the retention score (Table 6, with the same 512 GSM8K-FC decision-agent events as in Figure 3).

Retention scoreMedian ΔJ32Accuracy
Removal (AttuneKV’s r_i)0.0001782.4%
VATP style0.0002182.4%
Attention-only0.0003482.2%
Random0.0026181.8%

The three attention-based scores are close to one another, which is consistent with the observation in the original CAOTE paper, where the gain it adds on top of SnapKV is likewise small. We chose the removal criterion because it is the exact expression for the error of removing a single item, and because it can be computed directly from the same forward pass used for the recomputation ranking.

AuthorsOpenReview ↗

Response to reviewer zhmd (2/3)

Revisions to the paper: The contribution list in the Introduction now states the novelty as deciding recomputation and retention together at handoff time; Sections 2–3 cite CAOTE next to Eqs. (3) and (5); Appendix A.4 adds the retention-score comparison (Table 6); Section 5 spells out the differences of AttuneKV from KVShare, SnapKV, and CAOTE.

W2, Q1 · Speed and quality come from different workloads

Thank you for pointing this out. Measuring efficiency and accuracy separately in the original manuscript was a deliberate choice, and it is the same approach taken by KVCOMM and RelayCaching. In a multi-agent system, the messages each method reads are generated by upstream agents under that method, so their lengths differ, and TTFT depends not only on the algorithm but also on input length. When upstream messages are shorter, downstream TTFT also drops, so comparing the efficiency of the methods themselves requires all methods to read the same input. This is what Figure 6 does, and its fixed-length inputs are truncated messages with no gradable answer. To answer your question, we added a separate long-message experiment in which the four agents of the original GSM8K-FC workflow process question sets composed of 16 problems each. Each method runs the full workflow on its own (83 question sets, 1,319 answers in total, on an H100), and the table below places each method’s accuracy, input tokens, and TTFT side by side.

MethodAccuracyInput tokens (mean)TTFT (ms)
Dense77.10%10,911432.87
KVCOMM76.27%9,055185.67
RelayCaching76.35%8,674122.74
CacheBlend75.21%7,81792.44
AttuneKV76.95%9,715112.92

Answer to Q1: on this long-message workload, AttuneKV’s accuracy is 76.95%, 0.15 pp below Dense, while retaining only half of the message KV. Because input lengths differ across methods in the table, TTFT should be read together with input tokens, and speed comparisons between methods still rest on Figure 6, where inputs are identical.

The revised manuscript keeps the two measurements stated separately. Figure 6 has all methods read exactly the same input and sweeps the length from 512 to 12,288 tokens, to compare the efficiency of the methods. Table 3 answers the question of accuracy under long messages and lists each method’s actual input length and latency on its own workflow.

Revisions to the paper: Section 4 adds a Long-Message Handoff subsection (4.5) with Table 3, and Appendix B.7 gives the detailed setup of the workload.

Q2 · TTFT of native workflows

Thank you for the question. We have measured these numbers and list them in the table below. Across the nine workflows in the original Table 1, AttuneKV’s TTFT is 36.6–45.1 ms, and Dense’s is 38.5–130.8 ms.

DatasetSettingDense input tokensDense TTFT (ms)AttuneKV input tokensAttuneKV TTFT (ms)
GSM8KFC1,11146.41,35837.7
GSM8KInd.1,15746.81,15737.4
GSM8KCI23,725130.82,02140.0
MMLUFC1,25948.91,18637.6
MMLUInd.1,93871.21,93840.5
MMLUCI23,560119.42,92745.1
HumanEvalFC82538.597736.6
HumanEvalInd.1,19347.21,19337.5
HumanEvalCI21,22148.61,24237.8

As explained in the previous question, TTFT here depends not only on the algorithm but also on the length of the messages each method generates itself, so we list input tokens alongside TTFT. Efficiency comparisons between methods still rest on Figure 6, where all methods read the same input.

Revisions to the paper: Appendix B.6 adds the input length and TTFT of each workflow of the original Table 1 (Table 9), and Section 4.2 points to it.

AuthorsOpenReview ↗

Response to reviewer zhmd (3/3)

Q4 · KVCOMM’s anchor hit rate

Thank you for the question. This number should indeed be in the paper, so we list the hit rate at each length in Figure 6 in the table below. The hit rate falls as messages get longer, from 100% at 512 and 1K tokens to 60% at 12K. Each request has two messages, and a miss on either one triggers a full contextual prefill and an anchor write. At the 12K point, only one of the five requests hits on both messages, and its TTFT was 105 ms; each of the other four had one miss, with TTFT of 506–612 ms. The anchor configuration follows the authors’ public reference setting (20 anchors per slot, γ = 0.3). The number of anchors during measurement was at most 19, so no eviction was triggered.

Total message length (tokens)Messages hitRequests with both hitAnchors at measurementPool at measurement
51210/105/590.6 GiB
1,02410/105/560.7 GiB
2,0488/103/511–142.7–3.5 GiB
4,0969/104/55–62.5–3.0 GiB
8,1929/104/511–1411.0–14.0 GiB
12,2886/101/512–1918.0–28.5 GiB

Revisions to the paper: The KVCOMM paragraph in Section 4.3 adds how the hit rate changes with length, and Appendix B.4 lists the hit rate at each length and the TTFT of the two kinds of requests at the 12K point.

Q5 · MMLU size and the split for diagnostics

Thank you for the question. MMLU uses a fixed set of 153 questions, following the settings of the public KVCOMM and RelayCaching repositories. The 512 diagnostic events come from the GSM8K training split, which does not overlap with the test split used for evaluation in Table 1. The original manuscript did not state these two points. Thank you for the reminder; the revised manuscript adds them to Section 2 and Section 4.1.

W2 · Memory fairness, KVCOMM, and the abstract’s numbers

Thank you for pointing this out. We agree that the 2× corresponds directly to the ρ = 50% operating point, and since the baselines in the original manuscript all keep the full message KV, the number itself does not show whether the baselines could do the same. Thank you for pointing this out. What we want to claim is that AttuneKV can maintain accuracy while keeping only half of the message KV, and that deciding which half to keep requires no additional signal. CacheBlend and RelayCaching focus on recomputation and by design retain all of the message KV. For them to also keep only 50%, an additional probe after recomputation is needed to decide what to retain, which costs more time. In Figure 6(b) of the revised manuscript, we measure the time required to additionally apply AttuneKV’s retention on top of CacheBlend and RelayCaching. They need nearly twice as much time to decide which tokens to retain: AttuneKV adds 30 ms, whereas CacheBlend and RelayCaching each require an extra 60 ms or more.

The revised manuscript removes the 34× multiplier from the abstract and instead reports two kinds of memory separately. The first is the message KV per active handoff (KVCOMM 1.50 GiB, AttuneKV 0.75 GiB). The second is the anchor pool that KVCOMM must keep resident between requests, which is about 24 GiB in our experiments. The message KV retained by AttuneKV is released when the request ends, whereas the anchor pool is kept for reuse by later requests, so although amortization lowers the cost allocated to each request, this capacity must still be reserved at all times. The KVCOMM paper itself also lists memory as the price paid for its prefill speedup. The KVCOMM anchor configuration follows the authors’ public reference setting (as shown in Appendix B.2); at 12K tokens, one anchor is about 1.5 GiB, and the 24.0 GiB is the median of the anchor pool size over five measurements.

Both numbers in the abstract have been revised as you suggested. The revised abstract lists the speedups over Dense, RelayCaching, and CacheBlend side by side (2.92×, 1.45×, 1.12×). The revised manuscript lists all three together so that readers can compare them directly. We have also changed the quality statement to “the largest drop relative to dense prefill is 3.56 pp,” which occurs on GSM8K-FC.

Revisions to the paper: The numbers in the abstract and introduction, the memory description in Section 4.3 (reporting message KV and the anchor pool separately), and the new Table 3 in Section 4.5.

Figures: Fig. R1: Revised Figure 6; panel (b) adds CacheBlend and RelayCaching with AttuneKV’s retention step. link

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response: Summary of Revisions (1/2)

Thank you to the AC and the four reviewers for your rigorous and constructive reviews. Your comments helped us state our contributions more clearly and made our experiments more complete.

During the rebuttal period, we added new experiments on the positioning of the technical contribution, the comparison with prior work, and the measurement of accuracy and efficiency. The key points are summarized below.

1. Technical contribution (responding to zhmd, qFpn, i37j, g5uy)

  • Positioning of the contribution: Existing KV handoff methods (e.g., CacheBlend, KVCOMM, RelayCaching) focus on recomputation and retain all of the message KV, so their memory usage is never lower than that of dense prefill. We have clarified that AttuneKV is the first KV handoff method to study how to bring both TTFT and memory below dense prefill efficiently, achieving a 2.92× speedup on the same 12,288-token input while needing only half of the message KV (Figure 6).
  • One signal, two decisions: By running a forward pass on only the first suffix token after the message, we obtain, before any recomputation takes place, a ranking of both the positions to recompute and the positions to retain, from that token’s attention over the message KV. No message position needs to be forwarded in the process. When we widen the probe to the entire suffix, coverage increases by only 0.1 pp, showing that a single token already captures the information available at handoff time. The 12-point gap to the Oracle comes from information that does not yet exist at handoff time.
  • Cost of retention: We measured the time of the retention step on the same input (Figure 6(b)). AttuneKV’s retention shares the same probe as recomputation and adds only 30 ms, whereas attaching the same 50% retention to CacheBlend and RelayCaching requires obtaining another signal after recomputation, adding more than 60 ms. AttuneKV with retention takes 165 ms, still faster than CacheBlend keeping all message KV (186 ms), showing that halving memory costs AttuneKV almost no latency.
  • Accuracy: We added a long-message experiment (inputs averaging about 10K tokens, 1,319 answers in total, Table 3). With only half of the message KV retained, AttuneKV reaches 76.95% accuracy versus 77.10% for Dense, showing that retaining half of the message KV is sufficient to maintain task accuracy.

2. Comparison with prior work (responding to zhmd, qFpn, i37j)

  • Retention scores: At the same ρ = 50%, we compared removal, VATP, and attention-only, and the three achieve comparable accuracy (Table 6). We also now explicitly cite CAOTE next to Eqs. (3) and (5).
  • Source of the probe: We applied a ProphetKV-style query probe to the same handoff events. Its coverage is 27.7% versus 42.5% for AttuneKV, and obtaining the signal takes about four times as long, showing that a single token after the message already carries the information the receiver needs.

3. Measurement and statistics (responding to zhmd, qFpn, i37j)

  • TTFT on native workflows: We measured the input length and TTFT of the nine workflows in the original Table 1. AttuneKV’s TTFT is 36.6–45.1 ms and Dense’s is 38.5–130.8 ms.
  • KVCOMM: We now report memory separately as message KV and the anchor pool, and we added the anchor hit rate at each length, which is 60% at 12K tokens.
  • Statistical variance: We reran the three MMLU workflows with three seeds. AttuneKV vs. Dense: 66.2 ± 0.4 vs. 67.8 ± 2.9 (FC), 73.9 ± 1.3 vs. 71.9 ± 1.7 (Independent), and 71.0 ± 2.9 vs. 71.1 ± 1.6 (CI2), showing that the two perform comparably on MMLU.

Conclusion

We believe AttuneKV is the first KV handoff method to study how to bring both TTFT and memory below dense prefill efficiently. With these added experiments and revisions, the manuscript now addresses the reviewers’ main concerns.

AuthorsOpenReview ↗

General Response: Summary of Revisions (2/2)

Anonymous code. Our code and experiment configurations are available in an anonymous repository: https://anonymous.4open.science/r/attunekv-release-4E11/

Figures.

Fig. R1: Revised Figure 6; panel (b) adds CacheBlend and RelayCaching with AttuneKV’s retention step. link

Fig. R2: Breakdown of the decision agent’s subsequent attention in the long-message experiment (Figure 8 in Appendix B.7 of the revised manuscript). link

Fig. R3: Stability of the retention set over decoding steps; at each step the retention set is recomputed with that step’s query and its overlap with the set chosen at handoff is reported (Figure 7 in Appendix A.5 of the revised manuscript). link

Changes in the revised manuscript.

  1. The abstract and introduction report the speedups over Dense, RelayCaching, and CacheBlend side by side, add the accuracy of the long-message experiment, and state quality as the largest drop relative to dense prefill (3.56 pp); the introduction also states the time cost of the retention step, and the contribution list now states the contribution as deciding recomputation and retention together at handoff time.
  2. Sections 2–3 cite CAOTE next to Eqs. (3) and (5), state the conditions under which the score holds (single-entry removal, fixed query) together with the exact expression for joint removal, and add the definition of the Oracle and the fact that the diagnostic events come from the training split.
  3. Section 4 adds the long-message experiment (Section 4.5 and Table 3). Section 4.2 adds the MMLU results under three seeds. Figure 6(b) adds the time of CacheBlend and RelayCaching with the retention step. Section 4.3 reports message KV and KVCOMM’s anchor pool separately and adds how the anchor hit rate changes with length.
  4. Section 5 spells out the differences from KVShare, SnapKV, CAOTE, ProphetKV, and QCFuse, and cites ReToken and KV-PRM.
  5. The appendix adds the comparison with question tokens as the probe and the explanation of the Oracle (A.3), the comparison of retention scores (A.4, Table 6), the stability of the retention set during decoding (A.5, Figure 7), the MMLU results of eight methods under three seeds (B.5), the input length and TTFT of each workflow of the original Table 1 (B.6), and the setup of the long-message experiment, the definitions of the two retention variants, and the attention breakdown figure (B.7).
  6. Figure 2 is corrected, the probing cost in Appendix A.2 is re-measured on the H100, and all timings in the revised manuscript are measured on the H100.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.