acceptodds
Under review as a conference paper at ICLR 2027

Learning to Carry: KV Cache Eviction for Multi-Turn Vision-Language Models

Abstract

Vision-language models (VLMs) have advanced rapidly in joint visual and textual reasoning, and are now widely applied to multi-turn dialogue. However, their key-value (KV) cache grows with the conversation, increasing memory use and latency. Compressing this cache faces two challenges: eviction must anticipate unknown future queries, and retention must balance modalities despite image entries dominating the cache while receiving little current-turn attention. We propose CarryKV, a learned KV cache eviction method that decides at each turn boundary which entries to carry forward. CarryKV combines Structure-Aware Encoding (SAE), which exposes retention-relevant content and structural cues, with Sketch-Conditioned Selection (SCS), which selects entries complementary to a sketch of randomly retained cache entries. SCS also reserves a minimum image quota before filling the remaining budget by score. We train the selector through future-turn distillation, minimizing the KL divergence between the frozen VLM's predictions with full and retained caches on the following response. Training uses only 100 dialogues, takes under 25 minutes on one A100, and yields a single checkpoint reusable across budgets. Integrated into vLLM, CarryKV compacts the retained cache and physically frees evicted blocks. Across MMDU, MultiVerse, MMRC, and ConvBench on Qwen2.5-VL-7B and GLM-4.6V-Flash-9B, CarryKV maintains strong response quality under aggressive cache compression while reaching - the peak throughput of the full-cache baseline.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Token Pruning, Vision Language Model, KV Cache Eviction

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_xSq9OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper proposes a new KV cache compression algorithm for multi-turn vision-language tasks. The paper identifies two key challenges in multi-turn VLM settings: (1) the future relevance of cached tokens is unknown at the time of eviction, and (2) cache occupancy can be highly imbalanced across modalities. To address these issues, the authors jointly train SAE and SCS using a future-distillation objective, allowing the model to estimate which cache entries should be evicted at the end of each turn. Experimental results show that the proposed method achieves better performance-efficiency trade-offs than existing KV cache retrieval or eviction methods such as TriAttention.

Critical strengths
  • The paper clearly identifies an important challenge in multi-turn KV cache eviction: the relevance of a cached entry to future turns is inherently unknown at the time the eviction decision is made. The proposed future-distillation framework provides a principled way to incorporate future relevance into the eviction policy.

  • The proposed method is integrated into vLLM, demonstrating its practical applicability. Compared with the evaluated baselines, the method achieves favorable system-level efficiency in terms of throughput, TTFT, and wall-clock latency while maintaining strong task performance.

Critical weaknesses
  • Since eviction is performed only at turn boundaries, the proposed method does not evict KV entries continuously within a turn. As a result, its peak memory usage may be higher than that of methods that perform dynamic eviction throughout generation. This could become a meaningful limitation in multi-turn dialogue settings involving long responses, such as visual reasoning tasks with lengthy chain-of-thought or generated outputs.
  • The paper does not compare against TrimKV and DBTrimKV, which the authors themselves describe as the closest learned baselines. Such comparisons would be important for positioning the proposed method relative to prior learned KV eviction approaches. If reproducing these methods on the models used in the paper is infeasible due to training cost or implementation constraints, an alternative would be to evaluate CarryKV on the target models or experimental settings originally used by TrimKV and DBTrimKV.
Questions for authors
  • CarryKV seems to provide somewhat better efficiency under the current configuration with (\beta=128), but it would be useful to understand whether this advantage remains when increasing (\beta), for example to (\beta=256) or (\beta=512).
  • The selected range of (C) appears somewhat arbitrary. Would it be more appropriate to determine a method-specific maximum feasible cache budget, (C_{\max}), and compare the methods over their respective operating ranges? Also, How would the efficiency of CarryKV and other baselines change under this setting?
  • TriAttention is evaluated using its original eviction strategy, whereas CarryKV performs eviction at turn boundaries. What happens if TriAttention is adapted to perform turn-level eviction under the same protocol as CarryKV? Does CarryKV still maintain its advantage in both task performance and system efficiency under such comparison?
6 comments

Q3-1: Memory–Efficiency Trade-off under Long Responses

R3-1: CarryKV supports KV eviction at arbitrary time during generation. We adopt turn-boundary eviction by default to balance memory efficiency, inference latency, and generation quality. To investigate this trade-off, we evaluate different within-turn eviction intervals on 30 long-history samples.

Table R3-1: Trade-off between Eviction Frequency, Memory, Latency, and Quality.

Eviction IntervalAvg. Peak KV (MiB)↓E2E (s)↓EvictionsOverall↑
Turn-boundary116.1552.27416.17
12856.1553.557166.13
25660.1552.65486.16
51268.1552.41146.26
102484.1552.19926.13

The results reveal a clear trade-off between peak KV memory and eviction overhead. More frequent eviction reduces memory consumption but introduces additional compaction costs. Notably, eviction every 512 tokens reduces average peak KV memory by 41.3%, with only 0.26% higher latency while maintaining generation quality.

These findings highlight the flexibility of CarryKV: turn-boundary eviction minimizes repeated compaction, while periodic eviction enables substantial memory savings for long-response scenarios under tighter memory constraints.

Q3-2: Comparison with Learned KV Eviction Baselines.

R3-2: Since the original TrimKV does not provide a publicly released checkpoint for Qwen3-VL-4B-Instruct, we additionally compare CarryKV with DBTrimKV, a learned KV eviction method with an official checkpoint for this backbone, on MMDU.

Table R3-2: Quality and Inference Efficiency Comparison on MMDU.

MethodOverall↑TTFT (ms)↓TPS (tok/s)↑E2E (s/turn)↓
DBTrimKV7.045104.21585.1070.33
CarryKV7.14198.65664.0751.82

CarryKV improves MMDU Overall by 0.096 over DBTrimKV, achieving 13.5% higher throughput and 26.3% lower E2E latency. These results demonstrate the effectiveness of CarryKV in improving both generation quality and inference efficiency under the evaluated setting.

Q3-3: Sensitivity to Eviction Intervals.

R3-3: To examine the effect of eviction timing, we compare CarryKV with native TriAttention under identical eviction schedules on Qwen2.5-VL-7B, using the same 1,939 dialogue turns, KV budget (B=1024B=1024), and concurrency (C=128C=128).

Table R3-3: Throughput and Peak GPU Memory under Matched Eviction Schedules.

Eviction ScheduleThroughput (tok/s)↑Peak GPU (GiB)↓
128 tokens703.48 / 678.0971.070 / 71.023
256 tokens729.43 / 690.3171.071 / 71.023
512 tokens867.86 / 871.3370.065 / 71.023
Turn-boundary894.07 / 876.4671.071 / 71.025

Values are reported as CarryKV / native TriAttention.

CarryKV achieves higher throughput in three of the four matched eviction schedules, with gains of up to 5.67%. Under turn-boundary eviction, CarryKV improves throughput by 2.01% with comparable peak GPU memory. At the 512-token interval, it achieves comparable throughput while reducing peak GPU memory by 0.96GiB. These results demonstrate competitive serving efficiency under matched eviction schedules.

Q3-4: Maximum Concurrency and Efficiency Scaling.

R3-4: To address this concern, we determine the maximum feasible concurrency (Cmax⁡C_{\max}) for each method and compare throughput at both method-specific maximum and matched concurrency levels.

Table R3-4: Maximum Concurrency and Efficiency.

BackboneMethodCmax⁡C_{\max}↑Throughput↑
Qwen2.5-VL-7BFullKV64502.51
Qwen2.5-VL-7BCarryKV219842.57
Qwen2.5-VL-7BTriAttention219736.50
GLM-4.6V-Flash-9BFullKV133514.94
GLM-4.6V-Flash-9BCarryKV430578.83
GLM-4.6V-Flash-9BTriAttention407563.80

Cmax⁡C_{\max} is the largest verified feasible concurrency.

Table R3-5: Throughput at Matched Concurrency.

BackboneCCFullKVCarryKVTriAttention
Qwen2.5-VL-7B32446.52645.57610.90
GLM-4.6V-Flash-9B64559.55659.55611.01

Relative to FullKV, CarryKV achieves 3.42× the maximum verified concurrency on Qwen and 3.23× on GLM. On both backbones, CarryKV achieves the highest throughput when each method is evaluated at its own maximum verified concurrency. Under matched concurrency, CarryKV consistently outperforms both FullKV and TriAttention, demonstrating that its efficiency gains extend beyond increased concurrency capacity.

Q3-5: Comparison with TriAttention

R3-5: To ensure a fair comparison, we adapt TriAttention to CarryKV's turn-boundary eviction framework, keeping structural protection, image allocation, and physical KV compaction identical. The only difference is the scoring function.

Table R3-6: Framework-Matched Comparison with TriAttention.

MethodOverall↑Throughput (tokens/s)↑
CarryKV6.18842.57
Matched TriAttention6.09833.41

Under identical retention rules, CarryKV improves Overall by 0.09, achieves 1.1% higher throughput. These results indicate that learned scoring contributes to generation quality beyond the shared eviction framework while preserving competitive serving efficiency.

Response 1: My concerns are resolved.

Response 2: Is it possible to report your method on the target models & evaluation settings on TrimKV?

Response 3: My concerns are resolved.

Response 4: The reported values appear inconsistent with the original paper. Table 2 reports the largest completed concurrency for FullKV as 32 on Qwen2.5-VL-7B and 64 on GLM-4.6V-Flash-9B, whereas Table R3-4 reports 64 and 133, respectively. The original experiments also report out-of-memory failures at concurrency 64 for Qwen and 96 for GLM, so extending the search range alone would not explain this difference. Could you clarify whether the configuration changed between these experiments? If so, please specify the changes and explain their impact on the reported concurrency limits and throughput. If the settings were unchanged, how can these results be reconciled?

Response 5: My concerns are resolved.

Reviewer_pyasOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies key–value (KV) cache eviction in multi-turn vision–language model (VLM) conversations. The authors identify two main challenges: future queries are unknown, and images introduced early in a conversation may still receive attention many turns later; meanwhile, there is a mismatch between modality-level cache occupancy and current attention, with images occupying more than half of the cache while receiving very little attention in the current turn. To address these challenges, the authors propose CarryKV, a lightweight learned eviction method applied at turn boundaries. It combines structure-aware encoding (SAE), which encodes KV direction, magnitude, age, and role, with sketch-conditioned selection (SCS), which retains protected entries and a minimum image quota and scores candidate entries using a sketch constructed from a randomly retained subset. The selector is trained by minimizing the KL divergence between the predictive distributions for the next-turn response. Training requires only 100 conversations and can be completed within 25 minutes on a single A100 GPU. The method also implements block-level compaction and physical GPU memory reclamation in vLLM, achieving peak throughput of 1.37–1.75× that of full-cache inference across two models and four multi-turn benchmarks.

Critical strengths
  1. The empirical motivation is clear (Figure 1), highlighting the combination of high GPU memory occupancy, low current-turn attention, and long-range cross-turn dependencies associated with visual tokens in multi-turn VLMs. Performing cache eviction at turn boundaries and sharing eviction decisions across KV heads within each layer naturally aligns with paged memory management in modern inference frameworks.

  2. Training is inexpensive, and the system implementation is solid. The selector contains only 95K–161K parameters; with straight-through survival gating and implicit differentiation, training can be completed in under 25 minutes on a single GPU, and a single set of weights can be reused across different budgets. The vLLM V1 implementation performs actual physical block repacking and reclamation, substantially improving maximum concurrency and actual throughput.

  3. The experimental reporting is transparent. The paper provides paired-bootstrap 95% confidence intervals (Table 8), as well as detailed score breakdowns across individual evaluation dimensions (Appendix D).

Critical weaknesses
  1. The role of the random sketch is insufficiently analyzed, and the inference ablations confound multiple factors. In SCS, the sampling rate ρ\rho simultaneously controls the number of unconditionally retained random tokens ∣R∣|\mathcal{R}|, which reduces the budget mm available for score-based selection, and the number of samples used to construct the sketch ctc_t. In Tables 4 and 16, setting ρinfer=0\rho_{\text{infer}}=0—retaining no random tokens and setting ct=0c_t=0—actually outperforms the default ρinfer=0.10\rho_{\text{infer}}=0.10 on MMDU (6.252 vs. 6.177) and MultiVerse (48.94 vs. 48.90). The authors do not further analyze whether this improvement comes from eliminating the budget allocation to 10% random, potentially low-quality tokens or from zeroing the sketch itself. Moreover, the observation that the sketch is needed during training but removing it at inference yields better performance is somewhat counterintuitive. This makes the random subset R\mathcal{R} appear more like a form of stochastic budget perturbation or masking regularization during training than an effective complementary selection mechanism at inference.

  2. Beyond the structural rules, the incremental contribution of the learned module is not sufficiently substantial. CarryKV incorporates strong structural priors: protecting the system prefix, the current question, the tail of recent responses, and a minimum image quota BimgB_{\text{img}}. In Tables 4 and 16, retaining these rules while replacing learned scoring with purely random scoring (Random-fill) reduces the MultiVerse score by only 0.09 (48.81 vs. 48.90) and the MMDU score by only 0.25 relative to the full method. The module ablations in Table 3 also show that SAE+SCS yields an improvement of only +0.05 on Qwen2.5-VL-7B, which is within statistical uncertainty. Furthermore, under a tight budget on Qwen2.5-VL-7B, CarryKV performs comparably to the training-free baseline TriAttention on most metrics and slightly worse on MMRC (Tables 1 and 8).

  3. Some hyperparameter choices and inconsistencies between training and inference warrant further examination. First, the default policy of protecting a fixed nrec=256n_{\text{rec}}=256 recent response tokens consumes exactly half of the budget when B=512B=512. Tables 5 and 16 show that reducing this value to 64 yields clear improvements (+0.38 on MultiVerse and +0.125 on MMDU). Second, Appendix C.1 states that the minimum image quota BimgB_{\text{img}} is selected from a global pool during training but distributed evenly across resident images during inference. The paper provides no ablation analyzing this change in policy between training and inference.

Questions for authors
  1. To disentangle the effects of random-token budget consumption and sketch-conditioned scoring under ρinfer\rho_{\text{infer}}, what would the performance on MMDU and MultiVerse be if 10% of the entries were sampled at inference only to compute the sketch ctc_t, without forcing R\mathcal{R} to remain in the cache, and the entire non-protected budget B−∣P∣B-|\mathcal{P}| were allocated to top-kk selection based on sketch-conditioned scores?

  2. The w/o SCS ablation in Table 3 sets the sketch to zero but still retains the random subset R\mathcal{R}, while Table 5 only evaluates ρtrain∈{0.10,0.25,0.50}\rho_{\text{train}}\in\{0.10,0.25,0.50\}. How would a retrained model perform if random sampling and the sketch were completely removed during both training and inference, i.e., ρtrain=0\rho_{\text{train}}=0 and ρinfer=0\rho_{\text{infer}}=0?

  3. At inference, how much does performance on MMDU and MMRC differ between using the same globally pooled image-quota allocation as in training and distributing the quota evenly across resident images?

3 comments

Q2-1: Analysis of Random Retention and Sketch Conditioning in SCS.

R2-1: SCS uses random retention during training to expose the selector to diverse cache states, while the sketch summarizes sampled KV entries to guide the selection of complementary information. To disentangle their contributions, we conduct controlled training and inference ablations.

(1) Training Ablation. We retrain four variants while keeping the training objective and all other settings identical.

Table R2-1: Ablation of SCS Training Components.

Training RandomTraining SketchKL ↓NLL ↓Overall ↑
✓✓0.05911.59296.18
✓✗0.05831.59116.09
✗✓0.06281.59846.12
✗✗0.06221.59776.16

The full configuration achieves the highest Overall score, demonstrating a positive interaction between random retention and sketch conditioning during training.

(2) Inference Analysis. We independently vary forced random retention and sketch conditioning under the same KV budget. The sketch-only setting computes ctc_t from sampled entries without forcing their retention, allocating the full non-protected budget to score-based selection.

Table R2-2: Ablation of SCS Inference Components.

Random RetentionSketchMMDU ↑MultiVerse ↑
✗✗6.2548.94
✗✓6.2048.88
✓✗6.1448.84
✓✓6.1848.90

With random retention, sketch conditioning improves MMDU by 0.04 and MultiVerse by 0.06, demonstrating its benefit under randomly retained contexts. These results also reveal a trade-off between random retention and learned selection: allocating more budget to the latter favors aggregate scores, while sampled retention can preserve complementary evidence.

To further investigate this trade-off, we conduct a fine-grained analysis and find that sampled retention (ρ=0.1\rho=0.1) benefits historical visual and contextual evidence recovery in multi-turn multimodal dialogues. As shown in Tables R2-3 and R2-4, it achieves more wins on visual understanding and history-dependent questions in MMDU, while recovering overlooked visual and contextual evidence in MultiVerse.

Table R2-3: Fine-Grained Comparison on MMDU.

Evaluation CategoryWins (ρ=0.1\rho=0.1)Wins (ρ=0\rho=0)TiesWin Rate
Visual Perception52144468054.0%
Image Relationship Understanding55047562053.7%
Older Historical Images30725533154.6%
Multiple Images31426335654.4%

Table R2-4: Checklist Evidence Recovery on MultiVerse.

Checklist CategoryMissing (ρ=0\rho=0)Recovered (ρ=0.1\rho=0.1)
Visual Perception & Image-Text Consistency51484 (16.3%)
Memory & Context Understanding24752 (21.1%)

Together, these findings highlight the trade-off between learned importance-based selection and complementary evidence preservation, supporting the motivation behind SCS to adapt KV importance estimation to different retention contexts.

Q2-2: Incremental Contribution of Learned Scoring.

R2-2: Structural protection and learned scoring serve complementary purposes: the former safeguards essential context and multimodal coverage, while the latter identifies valuable KV entries within the remaining budget.

To isolate the contribution of learned scoring, we replace it with Random-fill or TriAttention while keeping all structural protection rules and eviction settings identical.

Table R2-5: Effect of Scoring Strategies under Matched Retention Rules.

Scoring StrategyMMDU Overall ↑MMRC ↑
Random-fill5.921.97
Matched TriAttention6.092.28
Learned6.182.25

Compared with Random-fill, learned scoring improves MMDU and MMRC, demonstrating its contribution beyond structural protection. Compared with matched TriAttention, learned scoring further improves MMDU by 0.09, while achieving comparable MMRC performance. These results show that learned scoring contributes to KV selection quality beyond the shared structural rules.

Q2-3: Recent-Token Protection.

R2-3: Protecting recent response tokens helps maintain conversational continuity when future queries are unknown, but introduces a trade-off between recent-token protection and learned score-based selection. We therefore evaluate different nrecn_{\mathrm{rec}} values under two KV budgets.

Table R2-6: Effect of Recent-Token Protection on MMDU.

Budgetnrec=64n_{\mathrm{rec}}=64nrec=128n_{\mathrm{rec}}=128nrec=256n_{\mathrm{rec}}=256
B=512B=5126.226.175.75
B=1024B=10246.266.306.18

The optimal nrecn_{\mathrm{rec}} increases from 64 to 128 as the budget doubles, corresponding to 12.5% of the KV budget in both settings. These results suggest that budget-aware recent-token protection better balances conversational continuity and historical information retention than a fixed protection size.

AuthorsOpenReview ↗

More replies

Q2-4: Further Analysis of Image Quota Allocation.

R2-4: During training, global pooling allows the selector to learn relative importance across visual tokens. At inference, however, future queries are unknown, and global pooling may discard entire images that become relevant later. We therefore distribute the image quota across resident images to preserve visual coverage.

To assess this design choice, we compare both inference allocation strategies using the same globally pooled training configuration.

Table R2-7: Effect of Inference-Time Image Quota Allocation.

Inference AllocationMMDU ↑MMRC ↑
Global Pool6.152.28
Per-image6.182.25

Per-image allocation improves MMDU by 0.03, while global pooling performs slightly better on MMRC by 0.03. This suggests a trade-off between visual coverage and importance-based selection, consistent with the more image-intensive interactions in MMDU and the relatively fewer images in MMRC. Per-image allocation favors coverage across images, whereas global pooling allows more flexible importance-based selection. Both strategies achieve comparable overall quality.

AuthorsOpenReview ↗

Follow-up on our Rebuttal

Dear Reviewer pyas,

Thank you for your thoughtful and constructive comments. We have carefully addressed your four main concerns and summarize the key additional evidence below:

  1. Contribution of SCS: We conduct controlled training and inference ablations to disentangle the contributions of random retention and sketch conditioning. We further provide fine-grained analyses demonstrating the benefits of SCS in recovering visual and contextual evidence, while clarifying its trade-off with learned selection.

  2. Learned Scoring: We compare CarryKV with Random-fill and matched TriAttention under identical settings, demonstrating the additional contribution of learned scoring beyond structural protection.

  3. Recent-Token Protection: We evaluate different numbers of protected recent tokens across KV budgets, showing that budget-aware protection better balances conversational continuity and historical information retention.

  4. Image Quota Allocation: We compare global pooling and per-image allocation on MMDU and MMRC, demonstrating the trade-off between preserving visual coverage and flexible importance-based selection.

As the discussion period progresses, we would greatly appreciate your feedback on whether these additional experiments and analyses have adequately addressed your concerns. If any questions remain, we would be glad to provide further clarification. We would also be grateful if you could reconsider your assessment in light of the additional evidence.

Thank you again for your time and consideration.

Best regards,
The Authors

Reviewer_nfFTOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies irreversible KV cache eviction for persistent multi-turn VLM conversations, where a retention decision must be made before the next user query is available. CarryKV uses a small shared network to encode KV content, token role, layer identity, and age, then scores entries conditioned on a sketch of a randomly retained subset. Structural protection and an image quota constrain the retained set. The selector is trained by distilling the frozen VLM's full-cache predictions on the following response, with a straight-through relaxation of selection. A vLLM implementation compacts retained entries and returns evicted blocks to the cache pool. Experiments cover two VLMs, four dialogue benchmarks, several budgets, and measured serving performance.

Critical strengths

The paper addresses a useful deployment problem with an economical solution. Deciding what to retain before the next query arrives is a meaningful setting for conversational VLMs, and the analysis in Figure 1 motivates why recent attention alone can undervalue historical images. The selector is small, requires only 100 training dialogues, and transfers across budgets and three benchmarks outside its training corpus without retraining. The evaluation generally uses the model's own generated history, allowing retention errors to propagate across turns; the fixed-history exception for MMRC is clearly documented. These choices make the complete recipe practically interesting.

The efficiency evidence is stronger than a theoretical cache-size argument. Table 2 and Figures 6 and 10 report throughput and latency in a common vLLM runtime, including scoring and compaction in the end-to-end measurements. Physical block reclamation increases feasible concurrency, while the quality results show that useful responses can survive substantial historical-cache compression. The appendices also improve the credibility of the evaluation by reporting paired confidence intervals, distinguishing sparse reads from physical eviction, and identifying settings where CarryKV does not outperform TriAttention. In particular, the GLM results provide evidence of quality improvements alongside the serving gains.

Critical weaknesses

The comparisons do not yet separate the benefit of the learned retention policy from the benefit of changing when and under what rules eviction occurs. CarryKV preserves the entire current input and response until the turn ends, whereas TriAttention evicts during generation whenever its cache exceeds the budget (Table 9). CarryKV also adds protected text and an image quota that is distributed across images at inference (Appendix C.1). Thus equal nominal budgets leave several consequential policy differences uncontrolled. This is visible on MultiVerse: Appendix D.1 attributes part of the advantage to preserving the first response before any eviction, and replacing learned scores with Random-fill changes the aggregate by only 0.09 points (Table 4). Random-fill is a useful control and establishes that learned scores help on Qwen MMDU, but it does not establish an advantage over an informed heuristic under the same boundary schedule and constraints. The peak throughput advantage over TriAttention is approximately 11% on Qwen and 7% on GLM, substantially smaller than the headline comparison with Full KV; different eviction frequencies could also contribute to this margin. A boundary-only TriAttention scorer using the same protection, image quotas, and compaction would be a focused way to test whether CarryKV improves the quality–throughput trade-off beyond this deployment recipe. Appendix E's similar peak pool occupancy does not resolve the timing and scoring confound.

The evidence for sketch-conditioned complementarity, a central methodological contribution, remains limited. Table 3 supports a modest combined benefit of SAE and SCS on GLM, but the combined gain on Qwen is not statistically significant, as Appendix B.3 acknowledges. More directly, setting the inference sampling ratio to zero preserves quality on both tested benchmarks (Table 4). By Equations 10–11, this sets the sketch and its interaction term to zero, so the resulting policy performs no sketch-conditioned comparison at deployment. This does not rule out a useful training effect, but the proposed explanation that random sketches act as a training device lacks the decisive control: training with no random subset and no sketch under the same future-turn objective. The existing training-ratio sweep uses only positive ratios, while the no-SCS ablation still reserves random entries. Consequently, the paper demonstrates a useful learned eviction recipe more clearly than it demonstrates why conditioning on a partially retained set is needed. A matched control would clarify whether the contribution is complementarity learning, training regularization, or primarily the lightweight entry scorer and structural rules.

Minor Clarification

Appendix D.3 says that the protected tail and image floor fix 416 of 512 entries before scoring, leaving fewer than 96 entries for the selector. The image floor fixes a modality count, but Equation 12 still uses learned scores to choose which image entries satisfy it. The statement should distinguish unrestricted budget slots from token identities selected by score; the Random-fill comparison is more informative about the scorer's contribution than this slot count alone.

Questions for authors
  1. How do quality and throughput change if TriAttention scores are used only at turn boundaries, with CarryKV's protected entries, per-image quotas, and cache compaction? This would help distinguish the value of learned scoring from the effects of eviction timing and structural protection.
  2. What happens when the selector is trained with both the random subset and sketch removed, while keeping the same training dialogues, future-turn loss, and structural constraints? Does sketch-based training improve over this simpler learned policy, especially when both are evaluated with deterministic inference?
  3. How much of the multi-image retention benefit comes from distributing the image floor across images at inference rather than using the pooled quota employed during training? A pooled-versus-distributed inference comparison on MMDU would clarify the contribution of this additional rule.
4 comments

Q1-1: Validating CarryKV's Gains through a Framework-Matched Comparison with TriAttention

R1-1: CarryKV performs eviction at turn boundaries, with retained KV entries determined jointly by structural protection and learned importance scores. To disentangle the contributions of the framework and learned scoring, we replace CarryKV's learned scorer with TriAttention's heuristic scorer while keeping all other components unchanged. The results are as shown below.

Table R1-1: Quality and Efficiency under Matched Retention Rules.

MethodOverall↑Throughput (tokens/s)↑
CarryKV6.18842.57
Matched TriAttention6.09833.41

CarryKV improves Overall by 0.09 over matched TriAttention, while achieving 1.1% higher throughput. These results demonstrate that learned scoring contributes to generation quality beyond the shared eviction framework while maintaining competitive serving efficiency.

Q1-2: Contribution of Sketch-Conditioned Learning.

R1-2: (1) Motivation. SCS conditions KV importance estimation on a randomly retained sketch and minimizes future-turn KL divergence from FullKV. This encourages the selector to preserve complementary information across diverse retention states rather than relying on independent importance scores.

(2) Training Ablation. To evaluate the contribution of SCS, we retrained four variants differing only in random retention and sketch conditioning, while keeping all other settings unchanged.

Table R1-2: Ablation of SCS Training Components.

Training RandomTraining SketchKL↓NLL↓Overall↑
✓✓0.05911.59296.18
✓✗0.05831.59116.09
✗✓0.06281.59846.12
✗✗0.06221.59776.16

With random retention, sketch conditioning improves Overall by 0.09, whereas sketch conditioning alone provides no gain. This supports the importance of their interaction in learning complementary KV importance.

(3) Inference Analysis. Although ρinfer=0\rho_{\mathrm{infer}}=0 achieves a slightly higher aggregate score, our fine-grained analysis shows that sampled retention (ρinfer=0.1\rho_{\mathrm{infer}}=0.1) can recover visual and contextual evidence overlooked by deterministic selection.

MMDU. We compare the two settings across visual understanding dimensions and image-dependent question categories.

Table R1-3: Per-Dimension Comparison on MMDU.

Evaluation CategoryWins (ρ=0.1\rho=0.1)Wins (ρ=0\rho=0)TiesWin Rate
Visual Perception52144468054.0%
Image Relationship Understanding55047562053.7%
Older Historical Images30725533154.6%
Multiple Images31426335654.4%

As shown above, ρ=0.1\rho=0.1 achieves more wins in visual perception, image relationship understanding, and questions involving older or multiple images, suggesting that sampled retention helps preserve complementary visual evidence across multi-turn interactions.

MultiVerse. We further analyze the evaluation checklists of MultiVerse to assess the recovery of overlooked visual and contextual evidence.

Table R1-4: Checklist Evidence Recovery on MultiVerse.

Checklist CategoryMissing (ρ=0\rho=0)Recovered (ρ=0.1\rho=0.1)
Visual Perception & Image-Text Consistency51484 (16.3%)
Memory & Context Understanding24752 (21.1%)

As shown in Table R1-4, ρ=0.1\rho=0.1 recovers previously missed visual details and contextual information, demonstrating the benefit of sampled retention in preserving complementary evidence across multi-turn interactions.

Q1-3: Contribution of Per-Image Quotas

R1-3: To assess the contribution of per-image quotas, we compare per-image allocation with the training-time global pooling strategy, keeping all other settings unchanged.

Table R1-5: Effect of Image Quota Allocation on MMDU.

Image AllocationOverall ↑Later Turns ↑Multi-Image Overall ↑
Global Pool6.156.196.3675
Per-image6.186.216.3767

Per-image allocation improves Overall by 0.03 and Later Turns by 0.02, with a slight improvement on multi-image questions. These results indicate that per-image quotas provide an additional benefit through balanced visual retention, while global pooling maintains comparable multi-image performance.

AuthorsOpenReview ↗

Clarification of Appendix D.3

Thank you for pointing this out. We have revised the manuscript in Appendix D.3 to distinguish reserved budget slots from score-selected token identities. Specifically, the image floor constrains only the number of retained visual entries, while their identities are determined by learned scores (Eq. 12). We further clarify the scorer's contribution through the Random-fill comparison under identical structural constraints.

AuthorsOpenReview ↗

Follow-up on Our Rebuttal

Dear Reviewer nfFT,

Thank you for your thoughtful and constructive comments. We have carefully addressed your three main concerns and summarize the key additional evidence below:

  1. Learned Scoring: We compare CarryKV with matched TriAttention to demonstrate the benefits of our learned scoring mechanism under identical settings.
  2. Contribution of SCS: We provide controlled ablations of random retention and sketch conditioning during training, and further demonstrate the effectiveness of SCS in recovering visual and contextual evidence during inference.
  3. Image-Floor Allocation: We compare pooled and per-image distributed image-floor allocation on MMDU to quantify the contribution of the per-image allocation rule during inference.

As the discussion period has passed its midpoint, we would greatly appreciate your feedback on whether these additional experiments and analyses have adequately addressed your concerns. If any questions remain, we would be glad to provide further clarification. We would also be grateful if you could reconsider your assessment in light of the additional evidence.

Thank you again for your time and consideration.

Best regards,
The Authors

Dear Reviewer nfFT,

Thank you again for your thoughtful review. We have directly addressed the raised concerns in the rebuttal. As the deadline for the discussion period is approaching, we sincerely ask that, if you have any additional comments or suggestions. We would be very glad if you could consider raising your score, according to the quality of the paper and the rebuttal. Thank you again for your time and consideration.

Best regards,

Submission 4856 Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Response to All Reviewers and Area Chairs: Summary of Rebuttal

Dear Reviewers and Area Chairs,

We sincerely thank all reviewers (R1-nfFT, R2-pyas, R3-xSq9) for their constructive feedback and positive assessments of CarryKV. We are pleased that:

  • All reviewers recognize the importance of the problem, particularly the challenge of preserving historical information when future queries are unknown.
  • Reviewers appreciate our lightweight and principled design, including future-turn distillation (R3), efficient training (R2), and practical retention mechanisms (R1).
  • All reviewers acknowledge our practical system contributions, including vLLM integration, physical KV compaction, and favorable performance–efficiency trade-offs.

We have carefully addressed each review and conducted additional experiments to strengthen our work. We summarize the key findings below:

  1. Effectiveness of learned KV selection. Controlled comparisons with Random-fill and framework-matched TriAttention demonstrate the contribution of learned scoring beyond structural protection, improving MMDU quality with comparable serving efficiency.

  2. Detailed ablation studies. Training and inference ablations clarify the roles of random retention and sketch conditioning in SCS. Fine-grained analyses further reveal their contributions to preserving historical visual and contextual evidence.

  3. Serving efficiency and scalability. We evaluate method-specific maximum concurrency, throughput, and inference latency, alongside matched comparisons under different eviction schedules.

  4. Comparison with learned KV eviction baselines. We additionally compare against DBTrimKV on Qwen3-VL-4B-Instruct. CarryKV achieves higher quality and throughput.

  5. Additional sensitivity analyses. We investigate KV budgets, recent-token protection, image quota allocation, and eviction intervals to better understand the trade-offs between generation quality and inference efficiency.

Collectively, these results provide further evidence for learning KV importance through future-turn supervision, demonstrating the effectiveness and practical value of CarryKV for multi-turn multimodal inference.

We appreciate the reviewers' feedback and hope our additional analyses and experiments address their concerns.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.