acceptodds
Withdrawn from ICLR 2027

Coverage Is Uniquely Cacheable: Cache-Once Coverage Compression for Multi-Turn Vision-Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Vision-language models (VLMs) are increasingly served in multi-turn conversation, where an image is encoded once and its visual key-value (KV) cache is reused across every subsequent turn. Visual token compression, however, is designed for single-pass inference. In multi-turn serving, current methods must either keep the full visual KV resident and re-select tokens for every new question, forfeiting the memory saving, or freeze a first-turn selection that is stale for every later question. We present C³ (Cache-Once Coverage Compression), a training-free method that addresses this challenge. Because the cached subset must serve questions that are unknown at commitment time, C³ abandons query relevance and instead spreads the retained tokens across the image feature space. This enables C³ to preserve a nearby token for whatever visual content a later question may target. The selection depends only on the image, so the cached subset is exactly what any later turn would select, and the whole dialogue runs on one compact visual KV. Across 8 benchmarks, 3 VLMs, and 3 KV budgets, C³ is the only method that avoids severe accuracy collapse across all benchmarks, models, and turns, cutting the persistent visual KV by 6.7x, achieving the lowest cumulative time-to-first-token as dialogues lengthen, and matching or beating the strongest full-KV re-pruning baseline on most benchmarks.

Primary area:
foundation or frontier models, including LLMs
Keywords:
vision-language models, visual token compression, KV cache compression, multi-turn inference, efficient inference

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Wpz7OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper targets visual token compression for multi-turn VLM serving. In this setting, the visual KV cache persists across turns, so query-conditioned pruning must either keep the full KV to re-select tokens every turn or freeze a first-turn selection that goes stale. The proposed C3 is training-free: at the first turn it applies farthest-point sampling to the LLM-input visual features, caches only the KV of the selected tokens, and reuses it for all later turns. The paper motivates this with a k-center worst-case argument. Validation uses three 7B VLMs on eight single-turn VQA benchmarks grouped into synthetic dialogues. C3 beats frozen query-conditioned baselines, cuts the persistent visual KV by 6.7×, and gives the lowest cumulative TTFT.

Critical strengths
  1. Relevant deployment question. Visual token pruning is usually evaluated single-pass. Framing multi-turn serving by its persistent visual-KV footprint is a useful lens and gives the work practical motivation.

  2. Simple and easy to deploy. C3 is training-free, needs no attention maps (so it works with FlashAttention), and pays its selection cost once per image. Its weight is limited, though: the method is identical to DivPrune, and a cached random subset performs nearly as well.

Critical weaknesses
  1. Outdated base models. Experiments use only Qwen2-VL, LLaVA-1.5, and LLaVA-OneVision, although Qwen3.5, Qwen3.6 and Qwen3.8 are now available. Qwen2.5-VL is even wired into the supplementary code but not reported. Please evaluate at least on Qwen3-VL and Qwen3.5 / Qwen3.6.

  2. A key comparison is missing. Every Re-Prune baseline is compute-matched, so it keeps fewer than 15% of tokens per turn. There is no every-turn query-conditioned re-pruning at an equal 15% token count. That arm is the accuracy upper bound needed to show what C3 sacrifices by ignoring the query.

  3. Limited novelty. The selection rule is identical to DivPrune; recasting max-min dispersion as k-center does not change the algorithm. A cached random subset is within 1–2.6 points of C3, so the coverage criterion itself contributes little.

  4. Flawed multi-turn setting. Earlier questions and answers are evicted from the cache, which breaks follow-up questions and coreference and would seriously hurt the user experience. Under this protocol C3 reduces to single-turn DivPrune. Please evaluate with history retained, on real multi-turn benchmarks such as MMDU and ConvBench.

Questions for authors
  1. Newer base models. Please report the main results on at least Qwen3.5, whose high-resolution inputs are exactly the regime C3 targets.

  2. Equal-token re-pruning. Please add every-turn query-conditioned re-pruning at the same 15% token count as C3, with the full KV resident. This would show how much accuracy C3 gives up by ignoring the query.

Reviewer_uTixOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper is about saving memory when a VLM is used in a multi-turn chat about a single image. The image gets encoded once, and its visual KV cache sticks around for every question that follows, so whatever is kept in that cache is paid for over the whole conversation. The authors argue that existing pruning methods like FastV and SparseVLM don't fit this setting well, because they choose tokens based on the current question. You either have to keep the full cache around so you can re-select for every new question, or you lock in the first-turn selection and live with it being wrong for later questions.

Their method, C3, sidesteps this by not looking at the question at all. On the first turn it picks a subset of visual tokens that are spread out across the image's feature space, using greedy farthest-point sampling. It then caches only those tokens for the rest of the conversation. The argument is that if you don't know what later questions will ask about, the best bet is to make sure every part of the image has a kept token close to it.

They test this on Qwen2-VL, LLaVA-1.5 and LLaVA-OneVision across eight benchmarks. Each image's questions are treated as a conversation, and earlier turns' text is dropped. At a 15% cache budget, C3 beats a frozen attention-based baseline everywhere, does better on average than re-pruning baselines given the same compute, and shrinks OneVision's visual cache from 181 MB to 27 MB.

Critical strengths
  1. A clear way to measure the multi-turn problem, and a useful finding about attention-based selection at high resolution.

The paper makes token pruning for multi-turn chat something you can measure. It defines when a selection can be reused across turns, measures the visual cache that persists for the whole conversation, and charges the one-time selection cost to the first turn. It also evaluates each question-dependent method in two forms: re-selecting every turn, and freezing the first-turn choice.

That comparison produces the paper's most useful result. At high resolution, committing to an attention-based selection can fail badly. On OneVision TextVQA, frozen attention-based methods score between 9.8 and 19.8, against 50.2 for C3. The authors note this is partly the attention criterion itself, not just the freezing. A question-independent selection degrades much more gracefully, and it can even be built before the first question arrives.

  1. A simple, deployable method with a transparent evaluation.

The method needs no training and no attention maps, so it works with. It runs once per image at modest cost. The evaluation covers three models, eight benchmarks and three budgets includes the serving harness and per-method configurations.

Critical weaknesses
  1. The paper's central dilemma doesn't hold for memory, and its efficiency claims depend on it.

a) The motivation rests on the idea that re-selecting tokens for each new question requires the full visual KV to stay resident and therefore saves no memory. But to re-select per question, you only need to keep what the recomputation starts from, such as the projected visual embeddings. For the Qwen2 based models, those take 12.5% of the full visual KV. On OneVision that is 22.6 MB versus 181 MB, which is actually smaller than C3's 27 MB.

b) Counted consistently, C3's advantage is either memory or per-turn compute, not both, and the paper doesn't cleanly measure either. The latency results don't seem to reflect actual cache reuse: Unpruned takes 200 ms on both the first turn and later turns, and C3's later turns cost exactly its first turn minus the selection time. Further cached random selection has a lower 8-turn cumulative latency than C3, so the lowest cumulative time-to-first-token claim doesn't hold.

  1. The method and setting largely already exist, and the experiments don't show that coverage is what drives the results.

The selection procedure is DivPrune's, as the authors acknowledge: its greedy coincides with the selection C3 commits at K=0. The claimed conceptual difference, a guarantee over discarded tokens that dispersion does not provide, doesn't hold. The multi-turn framing has precedents too, so the claim that prior work puts them to the same single-pass use isn't accurate.

  • VisionZip (cited) already argues that text-agnostic selection suits multi-turn KV reuse.
  • SparseVILA supplies the evaluation protocol.

On the empirical side, the main frozen baseline is the weakest frozen variant on every model. Other cacheable, question-independent selectors are missing on Qwen2-VL and OneVision. A random cached subset recovers about 86% of C3's gain over Frozen-attn on the ablation benchmarks.

Questions for authors
  1. Do later turns actually reuse the cached visual KV in the latency results ? Unpruned takes 200 ms on both the first turn and later turns, and C3's later turns (99 ms) cost exactly its first turn minus the selection time. That suggests the kept visual tokens are being re-prefilled each turn. If so, please say so, and report steady-state latency with real cache reuse for Unpruned, C3 and Random. This affects how I read the efficiency claims: with true reuse, C3's speedup over Unpruned would shrink, but its advantage over re-pruning could grow.

  2. Why are the re-pruning baselines charged the full visual KV? They already recompute every turn. Why couldn't they keep just the projected embeddings, which take 22.6 MB on OneVision, less than C3's 27 MB? If there's an implementation reason I'm missing, I'd like to understand it.

  3. Re-pruning from cached embeddings. Could you add SparseVLM and/or FastV re-pruning from cached embeddings at the same token count, charge them for embedding memory only, and measure latency with real cache reuse for the cached methods? If C3 matches or beats this at comparable memory with clearly lower per-turn latency, and I could consider raise my score.

Reviewer_vmdcOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies visual token compression in multi-turn visual question answering: visual tokens selected for the first question may not meet the information needs of subsequent questions as their focus changes. To address this issue, the authors propose C³, a training-free, question-independent compression method. The method uses image token features and farthest point sampling to select a subset that covers the feature space, then reuses the same visual KV cache across subsequent turns. The authors evaluate the method across multiple models, benchmarks, and cache budgets, demonstrating its question-answering performance and efficiency gains with reduced visual KV memory usage.

Critical strengths
  1. The paper addresses the tension between shifting question focus in multi-turn conversations and a fixed visual cache, extending the evaluation of compression beyond single-turn question answering to the availability of information in subsequent turns. This is a practically relevant problem and offers a valuable evaluation perspective for visual token compression.
  2. The method selects representative tokens using the image’s own features, enabling question-independent compression and cache reuse. The overall approach is simple and clear, and requires no additional training. Experiments across multiple models and tasks indicate some robustness under aggressive compression, particularly in certain high-resolution settings where it avoids the severe performance degradation observed in baseline methods.
Critical weaknesses
  1. A gap remains between visual feature representativeness and the semantic information required for conversation, and the paper may overstate the optimality of its coverage criterion. The method optimizes the coverage of visual tokens in feature space, but this does not necessarily amount to preserving the information most important for actual conversations. For example, a map may contain diverse visual content, such as terrain, rivers, and vegetation, while the user’s questions may focus primarily on transportation routes. Covering different types of visual content may not be the best allocation of a limited budget for question answering. Moreover, different roads with similar visual features may have distinct locations and connectivity, and therefore cannot simply substitute for one another. Section 3.2 introduces assumptions linking the evidence required by a question to feature distances, but the approximation guarantee concerns feature-space coverage distance rather than question-answering quality. The current theory and experiments are therefore insufficient to support a claim that the coverage criterion is universally optimal for unknown follow-up questions. The authors should further clarify the conditions under which this proxy objective is appropriate and its limitations.
  2. The experiments focus primarily on low cache retention ratios and provide limited analysis of the broader quality–memory trade-off and dynamic budgets.
  • The paper mainly reports results at B = 15%, and the budget sweep covers only 10%, 15%, and 25%. Within this range, some tasks already exhibit substantial performance losses relative to the uncompressed model. The existing results therefore do not establish whether the method retains its advantages under milder compression. I recommend evaluating higher KV retention ratios, presenting more complete curves of quality versus cache memory usage, and analyzing how sensitive different tasks are to budget changes.

  • From a systems perspective, multi-turn conversations also offer opportunities to adjust the compression budget over time. The paper requires a small visual cache to be fixed in the first turn, but does not sufficiently discuss the serving scenarios in which this constraint applies. For example, a system could initially retain more visual information and progressively compress it as the conversation grows or memory pressure increases, balancing early-turn answer quality against long-term cache memory usage. Such strategies do not necessarily resolve the difficulties caused by changing follow-up questions, but they merit discussion as alternatives. Since the paper clears the text of previous questions and answers at every turn, the current evaluation does not cover scenarios in which visual cache space is gradually reallocated to conversation history. This issue need not be a primary contribution of the paper, but the authors should clarify the scope of applicability of the fixed-budget setting.

  1. The contribution beyond prior work needs further clarification. The paper acknowledges that its core farthest point selection procedure substantially overlaps with DivPrune, but the differences in the actual algorithm, cache construction, and multi-turn serving behavior remain unclear. Reinterpreting an existing selection procedure through a coverage objective has some value, but the paper should distinguish its new theoretical perspective, serving setting, and algorithmic contributions. If the main advantage comes from multi-turn cache reuse, the authors should specify what additional mechanism is introduced beyond directly caching the tokens selected by DivPrune, and validate it through corresponding experiments. In addition, SparseVLM Frozen achieves slightly higher average scores than C³ on Qwen2-VL and LLaVA-1.5 in Appendix Table 5. This suggests that the method’s advantages depend on the model, task, or conversation turn, and these variations should be presented and explained more fully in the main text.
Questions for authors
  1. Could the authors provide more direct evidence or case studies linking feature coverage to the preservation of semantic information critical for question answering? For example, different methods could be compared in terms of whether they retain the evidence needed to answer follow-up questions, and the relationship between coverage distance and answer accuracy could be analyzed. Route connectivity in maps, counting repeated objects, or information at specific positions in tables could also help illustrate the method’s applicability and limitations.
  2. Could the budget experiments be extended to higher KV retention ratios, such as 40%, 50%, or 75%, with complete curves of quality versus cache memory usage for the method and the main baselines? It would be helpful to analyze whether the method’s advantages persist across budgets and which tasks exhibit a marked quality drop near a particular budget threshold.
  3. Could the authors further clarify the baseline implementations and presentation of results? In particular, what are the practical differences between directly caching DivPrune-selected tokens and C³? In Appendix Table 5, which tasks and turns account for the slightly higher average performance of SparseVLM Frozen than C³ on the two models? I recommend including these strong baselines in the main text and reporting aggregate results for the first turn and subsequent turns separately to show the additional gains more clearly.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.