- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3(+1)
Read review
Paper summary
This paper proposes a new KV cache compression algorithm for multi-turn vision-language tasks. The paper identifies two key challenges in multi-turn VLM settings: (1) the future relevance of cached tokens is unknown at the time of eviction, and (2) cache occupancy can be highly imbalanced across modalities. To address these issues, the authors jointly train SAE and SCS using a future-distillation objective, allowing the model to estimate which cache entries should be evicted at the end of each turn. Experimental results show that the proposed method achieves better performance-efficiency trade-offs than existing KV cache retrieval or eviction methods such as TriAttention.
Critical strengths
-
The paper clearly identifies an important challenge in multi-turn KV cache eviction: the relevance of a cached entry to future turns is inherently unknown at the time the eviction decision is made. The proposed future-distillation framework provides a principled way to incorporate future relevance into the eviction policy.
-
The proposed method is integrated into vLLM, demonstrating its practical applicability. Compared with the evaluated baselines, the method achieves favorable system-level efficiency in terms of throughput, TTFT, and wall-clock latency while maintaining strong task performance.
Critical weaknesses
- Since eviction is performed only at turn boundaries, the proposed method does not evict KV entries continuously within a turn. As a result, its peak memory usage may be higher than that of methods that perform dynamic eviction throughout generation. This could become a meaningful limitation in multi-turn dialogue settings involving long responses, such as visual reasoning tasks with lengthy chain-of-thought or generated outputs.
- The paper does not compare against TrimKV and DBTrimKV, which the authors themselves describe as the closest learned baselines. Such comparisons would be important for positioning the proposed method relative to prior learned KV eviction approaches. If reproducing these methods on the models used in the paper is infeasible due to training cost or implementation constraints, an alternative would be to evaluate CarryKV on the target models or experimental settings originally used by TrimKV and DBTrimKV.
Questions for authors
- CarryKV seems to provide somewhat better efficiency under the current configuration with (\beta=128), but it would be useful to understand whether this advantage remains when increasing (\beta), for example to (\beta=256) or (\beta=512).
- The selected range of (C) appears somewhat arbitrary. Would it be more appropriate to determine a method-specific maximum feasible cache budget, (C_{\max}), and compare the methods over their respective operating ranges? Also, How would the efficiency of CarryKV and other baselines change under this setting?
- TriAttention is evaluated using its original eviction strategy, whereas CarryKV performs eviction at turn boundaries. What happens if TriAttention is adapted to perform turn-level eviction under the same protocol as CarryKV? Does CarryKV still maintain its advantage in both task performance and system efficiency under such comparison?
6 comments
Q3-5: Comparison with TriAttention
R3-5: To ensure a fair comparison, we adapt TriAttention to CarryKV's turn-boundary eviction framework, keeping structural protection, image allocation, and physical KV compaction identical. The only difference is the scoring function.
Table R3-6: Framework-Matched Comparison with TriAttention.
| Method | Overall↑ | Throughput (tokens/s)↑ |
|---|---|---|
| CarryKV | 6.18 | 842.57 |
| Matched TriAttention | 6.09 | 833.41 |
Under identical retention rules, CarryKV improves Overall by 0.09, achieves 1.1% higher throughput. These results indicate that learned scoring contributes to generation quality beyond the shared eviction framework while preserving competitive serving efficiency.
Response 1: My concerns are resolved.
Response 2: Is it possible to report your method on the target models & evaluation settings on TrimKV?
Response 3: My concerns are resolved.
Response 4: The reported values appear inconsistent with the original paper. Table 2 reports the largest completed concurrency for FullKV as 32 on Qwen2.5-VL-7B and 64 on GLM-4.6V-Flash-9B, whereas Table R3-4 reports 64 and 133, respectively. The original experiments also report out-of-memory failures at concurrency 64 for Qwen and 96 for GLM, so extending the search range alone would not explain this difference. Could you clarify whether the configuration changed between these experiments? If so, please specify the changes and explain their impact on the reported concurrency limits and throughput. If the settings were unchanged, how can these results be reconciled?
Response 5: My concerns are resolved.
Q3-1: Memory–Efficiency Trade-off under Long ResponsesR3-1: CarryKV supports KV eviction at arbitrary time during generation. We adopt turn-boundary eviction by default to balance memory efficiency, inference latency, and generation quality. To investigate this trade-off, we evaluate different within-turn eviction intervals on 30 long-history samples.Table R3-1: Trade-off between Eviction Frequency, Memory, Latency, and Quality.
The results reveal a clear trade-off between peak KV memory and eviction overhead. More frequent eviction reduces memory consumption but introduces additional compaction costs. Notably, eviction every 512 tokens reduces average peak KV memory by 41.3%, with only 0.26% higher latency while maintaining generation quality.
These findings highlight the flexibility of CarryKV: turn-boundary eviction minimizes repeated compaction, while periodic eviction enables substantial memory savings for long-response scenarios under tighter memory constraints.
Q3-2: Comparison with Learned KV Eviction Baselines.R3-2: Since the original TrimKV does not provide a publicly released checkpoint for Qwen3-VL-4B-Instruct, we additionally compare CarryKV with DBTrimKV, a learned KV eviction method with an official checkpoint for this backbone, on MMDU.Table R3-2: Quality and Inference Efficiency Comparison on MMDU.
CarryKV improves MMDU Overall by 0.096 over DBTrimKV, achieving 13.5% higher throughput and 26.3% lower E2E latency. These results demonstrate the effectiveness of CarryKV in improving both generation quality and inference efficiency under the evaluated setting.
Q3-3: Sensitivity to Eviction Intervals.R3-3: To examine the effect of eviction timing, we compare CarryKV with native TriAttention under identical eviction schedules on Qwen2.5-VL-7B, using the same 1,939 dialogue turns, KV budget (B=1024), and concurrency (C=128).Table R3-3: Throughput and Peak GPU Memory under Matched Eviction Schedules.
Values are reported as CarryKV / native TriAttention.
CarryKV achieves higher throughput in three of the four matched eviction schedules, with gains of up to 5.67%. Under turn-boundary eviction, CarryKV improves throughput by 2.01% with comparable peak GPU memory. At the 512-token interval, it achieves comparable throughput while reducing peak GPU memory by 0.96GiB. These results demonstrate competitive serving efficiency under matched eviction schedules.
Q3-4: Maximum Concurrency and Efficiency Scaling.R3-4: To address this concern, we determine the maximum feasible concurrency (Cmax) for each method and compare throughput at both method-specific maximum and matched concurrency levels.Table R3-4: Maximum Concurrency and Efficiency.
Cmax is the largest verified feasible concurrency.
Table R3-5: Throughput at Matched Concurrency.
Relative to FullKV, CarryKV achieves 3.42× the maximum verified concurrency on Qwen and 3.23× on GLM. On both backbones, CarryKV achieves the highest throughput when each method is evaluated at its own maximum verified concurrency. Under matched concurrency, CarryKV consistently outperforms both FullKV and TriAttention, demonstrating that its efficiency gains extend beyond increased concurrency capacity.