acceptodds
Under review as a conference paper at ICLR 2027

Fund Regions, Not Tokens: Query-Conditioned Budget Allocation for Extreme KV-Cache Eviction in Vision-Language Models

Abstract

Vision-language models (VLMs) need thousands of visual tokens for multi-page documents and high-resolution charts. The per-request key–value (KV) cache limits how many requests a device can serve concurrently. Existing eviction methods from text LLMs keep the global top-k tokens of a one-dimensional sequence. Visual evidence, however, occupies contiguous two-dimensional regions, and a region stays usable only if enough of its tokens remain. At extreme budgets of 1–3%, a global ranking spreads the cache over many locations and leaves each region too few tokens. To this end, we propose CARVE, which treats KV eviction as query-conditioned budget allocation to regions. CARVE partitions visual tokens into layout-aware cells, commits most of the budget to a few query-ranked regions, and selects tokens within them. When selection-time diagnostics find this allocation unsuitable, CARVE falls back to attention-based selection, and its gated mode CARVE-G to conservative token-level eviction. Experiments cover four models and four benchmarks at equal KV budget. On DocVQA at a 1% budget, CARVE retains 74.4% of the full-cache score on InternVL3-8B, where the strongest text-guided baseline retains 40%. CARVE-G stays within 1.6 points of its text-guided fallback in all sixteen model–budget settings.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Vision-Language Models, KV Cache Compression, Efficient Inference, Query-Conditioned Budget Allocation, Region-Aware Token Selection

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_F5ZeOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

CARVE partitions visual tokens into layout-aligned cells, ranks cells using query-conditioned token relevance, and reserves most of the cache budget for tokens within selected cells. Selection-time diagnostics can switch to attention-based selection; CARVE-G adds a confidence gate that falls back to the underlying token-level method. Compression occurs after prefill and before decoding, with a per-layer context budget of B=⌈ρN⌉B=\lceil\rho N\rceil.

The paper evaluates four VLMs on four visual question-answering benchmarks at cache retention ratios of 1%1\%, 3%3\%, 5%5\%, and 10%10\%. On InternVL3-8B DocVQA at 1%1\% retention, CARVE achieves 66.8 ANLS, compared with 35.5 for TGV-KV and 89.8 for the full cache. A stylized retrieval analysis motivates concentrating the budget when region rankings are reliable and the budget is small.

Critical strengths
  1. Substantial improvements under a controlled core comparison. CARVE uses the same base relevance signal as TGV-KV and the same total cache budget. The large DocVQA gains therefore provide meaningful evidence for changing the selection policy rather than merely improving token scoring (pp. 5–7, Table 1).
  2. Ablations directly probe allocation and geometry. The comparison with one-dimensional selection, uniform allocation, and random allocation is complemented by an experiment that shuffles cell membership while preserving the budget and cell count. The resulting degradation supports the importance of spatially coherent groups (pp. 7–8, Table 2a and Section 6.1).
  3. Broad evaluation within the stated task family. Four models, four benchmarks, and four budgets provide a reasonably informative picture of the method's operating regimes. The paper reports cases where other methods remain stronger.
  4. The gated variant addresses a real failure mode. CARVE can regress at larger budgets, and CARVE-G substantially reduces these regressions. Its worst single-benchmark difference relative to TGV-KV is reported as −1.6-1.6 points, rather than being presented as a universal improvement (Tables 1–2).
Critical weaknesses
  1. The full method relies on several insufficiently isolated heuristics. These include the attention/text signal switch, a micro-budget fallback, a 35-token post-image prompt threshold, multiple confidence thresholds, and hybrid scoring. The current ablations do not fully establish the necessity and portability of each rule (pp. 16–18, Appendix B).
  2. The theoretical result does not directly characterize the implemented algorithm. The analysis assumes a single evidence region and a concave retrieval utility, with its detailed reliability characterization restricted to two regions. The implementation instead uses max-pooled scores, discrete region selection, and fixed heuristics. The paper acknowledges that entropy, Gini, and top-mass statistics are not formally linked to ranking correctness (pp. 14–16).
  3. Context-cache retention is not total-memory reduction. The model still processes the full input during prefill, and newly generated tokens are appended afterward. Evaluation limits generation to 32 tokens. Thus, 1%1\% retained input-context entries should not be interpreted as 1%1\% total GPU memory or a demonstrated long-generation operating point (pp. 3 and 9).
  4. The practical efficiency evidence is incomplete. Decode latency is similar across methods, while CARVE and CARVE-G add approximately 30 ms and 60 ms of routing overhead per sample on InternVL3. Concurrent serving throughput, peak memory, and realized request capacity are not measured. Evaluation also remains single-image and short-answer despite the broader long-document motivation (p. 9).
Questions for authors
  1. Can you report a systematic ablation of the prompt-length fallback, micro-budget fallback, signal switch, and confidence gate across all four models? Which components account for most of the improvement?
  2. How were the development probes used to choose the routing thresholds separated from the final evaluation examples? How well do these thresholds transfer to longer questions and different prompt templates?
  3. Can you report actual KV-cache bytes, end-to-end latency including prefill and routing, and concurrent serving throughput at matched answer quality?
  4. How sensitive is the method to cell granularity and to questions requiring evidence from several spatially separated regions? Does the advantage persist with longer generated answers?
4 comments
AuthorsOpenReview ↗

Response to Reviewer F5Ze (1/2)

We thank you for the constructive assessment. Your request to establish the necessity of each rule was especially valuable. The systematic ablation it prompted showed that the prompt-length switch contributed nothing at 1-3% and made the selection depend on question length, so we removed it, and it clarified what the remaining rules do.

W1 / Q1. Heuristics.

We now report a systematic ablation on all four models (Table R8). Each entry is the change in the four-benchmark mean when one component is removed from (or, for the gate, added to) CARVE without the prompt-length switch, and the two prompt-length rows compare the submitted CARVE with and without the switch (full sets).

Table R8. Component ablation.

ComponentInternVL3-8BLLaVA-NeXTQwen2.5-VL-7BQwen3-VL-8B
Region allocation (random funded cells instead, signal switch off in both, 1%)-16.0-9.4-15.3-7.5
Prompt-length switch (removing it, 1%)+1.8+0.6+1.3+0.4
Prompt-length switch (removing it, 10%)-2.30.0-0.1+0.1
Micro-budget switch (removing it, 1%)+0.5-5.2+0.9+0.8
Signal switch (removing it, 1%)-3.3-0.5+0.50.0
Gate (adding it, 1%)-3.4-0.5-0.50.0
Gate (adding it, 10%)+10.30.0-0.1+0.2

The region allocation carries the gain. The signal switch matters on InternVL3-8B ChartQA (53.0 → 66.8). The micro-budget switch matters when the per-layer budget falls to about a dozen entries (LLaVA-NeXT at 1%), and we keep it because it prevents a 5.2-point loss there while costing at most 0.9 points elsewhere. The gate protects the loose budgets (49.1 → 82.9 on InternVL3-8B ChartQA at 10%), while CARVE is the mode for 1-3%.

W2. Theory.

The analysis is indeed stylized. It motivates concentration and the use of reliability diagnostics, and does not characterize the implemented selection. The revision states its assumptions next to the result (Appendix A) and presents it as motivation. Its uninformative endpoint is also visible in the corrected ablation, where random allocations do not beat uniform spreading at 1% (Uniform-2D 48.1 / 55.5, Random-2D 47.6 / 55.5 and random funded cells in CARVE 45.4 / 52.2 on InternVL3-8B / Qwen2.5-VL-7B in Table R1 of our answer to Reviewer Dnaf).

W3. Context-cache retention versus total memory.

The revision makes the wording precise. A ratio of 1% refers to the context cache retained after prefill, while prefill processes the full input and generated tokens are appended. It reports bytes explicitly (Table R4 below and Table 7 of the revision) and makes no claim about total or peak memory.

W4 / Q3. Efficiency.

Table R4 gives the KV bytes, the measured concurrency each ratio admits and the resulting decode throughput.

Table R4. Context-cache size and serving capacity (copied from our response to Reviewer e9BP for your convenience).

Visual contextFull cache per requestMax concurrent requests, 100% / 10% / 3% / 1%Decode throughput at max concurrency (tok/s)
1 high-resolution page (4.3K tokens)0.23 GiB73 / 678 / $\geq$1024 / $\geq$10241,317 / 4,548 / 5,948 / 6,218
8 images (34K tokens)1.8 GiB9 / 92 / 300 / 840181 / 1,617 / 3,397 / 5,071
32 images (131K tokens)7.0 GiB2 / 24 / 80 / 23544 / 452 / 1,454 / 3,333

(Measured on one 32 GB GPU with the language model of InternVL3-8B, which leaves about 17 GiB for the context cache after the weights, with 32 greedy decoding steps and batch size searched up to 1,024.)

Going from 3% to 1% raises decode throughput by 1.5×1.5\times for an eight-image context and 2.3×2.3\times for a 131K-token context. This serving gain holds for any eviction policy, and CARVE changes the accuracy at that budget, including on multi-page inputs. With 41 entries per layer, it keeps 92% of the full-cache ANLS on MP-DocVQA against 70% for TGV-KV (Table R6 in our answer to Reviewer e9BP). At near-matched quality (within 0.7 points), CARVE needs a third of the cache, scoring 77.5 on four pages with 41 entries per layer against 78.2 for TGV-KV with 123 (Table R5 in the same answer).

Latency. Prefill processes the full input for every method. On InternVL3-8B at 1% with batch size 1, the routing adds 30 ms per sample for CARVE and 60 ms for CARVE-G (7% and 14% of the prefill forward pass), and every method decodes at 25-26 ms/token (Section 6.5 of the revision and our answer to Reviewer Dnaf, Q4).

AuthorsOpenReview ↗

Response to Reviewer F5Ze (2/2)

Q2. Development probes and transfer.

The detection constants were set on the first 128 questions of the InternVL3-8B ChartQA test and DocVQA validation splits, so these questions are part of the evaluation sets. Excluding these questions changes CARVE by at most 0.7 points (DocVQA 69.5 → 69.3, ChartQA 66.8 → 67.5 at 1%) and TGV-KV by at most 0.3, so every comparison is unchanged. CARVE also transfers to longer questions and other templates. With a longer instruction (+15 tokens), it goes from 69.5 to 70.0 ANLS on InternVL3-8B DocVQA at 1% and from 78.4 to 77.6 on Qwen2.5-VL-7B, where the full cache itself drops by 2.2 points. With an alternative phrasing of the same length, it goes from 69.5 to 70.6 on InternVL3-8B DocVQA, and on ChartQA either template raises it from 66.8 to 69.5. The prompt-length switch we removed would instead have reverted every prompt of the longer template to the base selection (34.4 and 39.7 ANLS).

Q4. Cell granularity, multi-region evidence, longer answers.

  • Granularity. Accuracy stays within 1.5 points for R between 64 and 128 cells and drops for very fine cells (1.6 to 5.8 points lower at R=256R = 256 on DocVQA and ChartQA), where a funded cell holds too few tokens to cover an answer region.
  • Multi-region evidence. InfoVQA annotates answer types and operations. At 1% on InternVL3-8B, CARVE improves most on single-span answers (44.1 → 59.3 ANLS over TGV-KV), less on multi-span answers (20.7 → 22.2) and on questions requiring comparison, arithmetic or counting (46.9 → 51.9). Multi-span answers remain hard for every method at 1%.
  • Longer answers. Our benchmarks follow the short-answer protocol (≤32 generated tokens), so we make no claim about long generations. The retained context is fixed after prefill and generated tokens are appended for every method, so a longer answer draws on the same retained evidence.

Thank you for the detailed response and additional experiments. The rebuttal strengthens the paper and addresses several of my concerns.

The corrected allocation controls are particularly helpful. Query-ranked allocation retains a substantial advantage after correcting Uniform-2D’s tie-breaking, and replacing query-ranked cells with randomly selected cells within the otherwise unchanged CARVE pipeline causes large performance drops. This provides stronger evidence that query-conditioned region selection drives the gains. I also appreciate the removal of the prompt-length switch and hybrid scoring, together with the component ablations and question-rewrite experiments. The new MP-DocVQA results and the four-page comparison at near-matched quality make the practical motivation more convincing.

The response has sufficiently addressed my questions for this discussion. My overall assessment remains that CARVE makes a useful empirical contribution through query-conditioned region allocation under extreme post-prefill cache budgets. The revised controls and simplified routing policy provide a clearer account of why it works. I regard the theoretical analysis as useful motivation, and assess the implemented method on the strength of its empirical evidence.

Taking the contribution and its scope together, I remain supportive of acceptance at the level of weak acceptance and will maintain my rating of 3. The additional results strengthen this recommendation. My rating reflects my overall assessment of the paper’s contribution, with the strongest evidence concerning short-answer visual QA under severe context-cache constraints. I have no further questions at this stage.

AuthorsOpenReview ↗

Response to Reviewer F5Ze - Thank You

We thank you for the careful follow-up and for maintaining your support. Your request to justify each rule led to the systematic ablation and to the removal of the prompt-length switch, and your questions on serving capacity and multi-page inputs made the case for the 1-3% regime concrete. We agree with your reading of the scope: the evidence is strongest for short-answer visual QA under severe context-cache budgets, which is the scope our Limitations paragraph states.

Reviewer_e9BPOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper tackles KV-cache eviction for vision-language models at extreme budgets (1–3%), arguing that global token-level top-k scatters the budget and leaves no region legible. It proposes CARVE, which partitions visual tokens into 2D cells, commits most of the budget to a few query-ranked cells, and selects tokens within them, with a gated variant (CARVE-G) that falls back to conservative token-level eviction when region evidence is unreliable. Across four VLMs and four VQA benchmarks, CARVE yields large gains at extreme budget.

Critical strengths
  1. The writing is clear and easy to follow.

  2. The method is deployment-friendly.

Critical weaknesses
  1. I have a concern about the motivation of this problem setting. The introduction section doesn't explain why compression to 1-3% is needed. Which scenarios have this requirement? How much actual resource saving does it bring? Is the saving worth the accuracy drop? In which scenarios would the system still be usable with around 20-30% accuracy drop? The authors should make these points explicit in the introduction. Otherwise, such extreme compression may be an artificial need.

  2. The method depends on many hyperparameters. But I cannot find experiments about how to set these hyperparameters (like R, f, ω\omega, ...) or why the authors chose these values. It is hard to judge how sensitive the method is to these settings. Are the reported results tuned on a small set of specific hyperparameters? This leads to my concern about the robustness of this method when used in practice.

  3. The authors repeatedly claim that "the evidence needed to answer a question occupies a contiguous 2D region of the image". However, from my understanding, the token selection still follows a top-k rule. Cells and tokens in cells are also chosen by score. Nothing seems to guarantee that retained tokens are spatially contiguous. Could you clarify this and provide some theoretical analysis or qualitative visualizations?

  4. The evaluation is conducted on single images. Would this method also be effective in multi-image scenarios like documents?

Questions for authors

See weaknesses.

4 comments
AuthorsOpenReview ↗

Response to Reviewer e9BP (1/2)

We thank you for the careful reading and for pressing us on the motivation. Your question of why 1-3% matters motivated new experiments that make the practical relevance of this setting concrete (fixed-memory multi-page experiments, including real multi-page documents, and measured serving concurrency), and your remark on contiguity caught an imprecise claim that we now correct. We address your four points below, and the revised introduction states this motivation.

W1. Why 1-3%, and what it saves.

The context cache is what a request holds in GPU memory for the whole decode, and it grows linearly with the visual context. For InternVL3-8B (28 layers, 4 KV heads, head dimension 128, fp16) this is 56 KiB per token. For a single 4.3K-token image the full cache is only 0.23 GiB, so extreme ratios pay off when visual contexts are long or requests are many, as Table R4 shows.

Table R4. Context-cache size and serving capacity.

Visual contextFull cache per requestMax concurrent requests, 100% / 10% / 3% / 1%Decode throughput at max concurrency (tok/s)
1 high-resolution page (4.3K tokens)0.23 GiB73 / 678 / $\geq$1024 / $\geq$10241,317 / 4,548 / 5,948 / 6,218
8 images (34K tokens)1.8 GiB9 / 92 / 300 / 840181 / 1,617 / 3,397 / 5,071
32 images (131K tokens)7.0 GiB2 / 24 / 80 / 23544 / 452 / 1,454 / 3,333

(Measured on one 32 GB GPU with the language model of InternVL3-8B, which leaves about 17 GiB for the context cache after the weights, with 32 greedy decoding steps and batch size searched up to 1,024.) For a single page, going from 10% to 1% raises throughput by 37%, and for eight and for 32 images it raises throughput by 3.1×3.1\times and 7.4×7.4\times. When a deployment fixes the memory per request, longer visual inputs translate directly into smaller retention ratios, and multi-page and multi-image inputs land at 1-3%.

Is the saving worth the accuracy drop? Under a memory constraint, the alternative is not the full cache but another way to fit the same memory:

  • Another eviction policy at the same budget. At 1%, CARVE retains 72-82% of the full-cache four-benchmark mean where the strongest baseline on each model retains 64-71% (Table 1 of the revision).
  • A larger budget for the baseline. On Qwen2.5-VL-7B, CARVE at 3% (93.4% of the full-cache mean) exceeds every baseline at 5%.
  • A smaller input. Downscaling trades resolution for retention. At the same 41 entries per layer, a 1K-token page scores 84.4 with CARVE and 70.6 with TGV-KV, a 4K-token page 78.4 and 59.0. Shrinking further does not pay. The same pages shrunk to about 400 visual tokens and kept at 10%, about the memory of the original at 1%, score 76.3 on DocVQA and 59.9 on InfoVQA with CARVE (68.1 and 57.1 with TGV-KV), against 78.4 and 72.3 for CARVE at 1% on the original input. The 1-3% regime targets inputs whose token count cannot be reduced, such as multi-page documents.
  • A milder ratio. At 3% (a 33×33\times smaller context cache), CARVE retains 87-93% of the full-cache mean on all four models.

In the setting that motivates 1-3%, multi-page inputs under a fixed memory per request, the loss on Qwen2.5-VL-7B is 6-8% rather than 20-30%. With 41 entries per layer, CARVE keeps 93-94% of the full-cache score as pages are added (Table R5 below) and 92% on real multi-page documents (Table R6).

The scenario. Each DocVQA question is answered from its own page plus kk pages of other documents in random order (Qwen2.5-VL-7B, about 1K tokens per page, full DocVQA validation set of 5,349 questions), with the number of retained entries per layer held fixed, so the retention ratio falls as pages are added.

Table R5. DocVQA plus kk pages at fixed memory (Qwen2.5-VL-7B).

PagesFull cacheTGV-KVH2OSnapKVCARVECARVE-G
41 entries per layer ($\approx$4% → 1%)
190.170.654.537.884.483.8
287.059.743.736.480.779.4
482.951.937.134.877.575.5
123 entries per layer ($\approx$12% → 3%)
190.187.471.076.986.186.9
287.083.356.265.982.383.0
482.978.244.759.577.277.7

(Best compressed result per row in bold.) At 41 entries, TGV-KV falls from 78% to 63% of the full-cache score as pages are added. A single 4K-token page at 1% keeps the same 41 entries per layer as the four-page inputs, so our single-image benchmarks pose the same retention problem in a controlled form, with the same ordering of methods. Qwen3-VL-8B shows the same ordering with a larger loss (with four pages at 41 entries, CARVE scores 70.4, TGV-KV 56.2 and the full cache 87.7). With 123 entries, TGV-KV, CARVE and CARVE-G are within 1.3 points of each other, so the allocation matters most when only a few dozen entries per layer remain.

AuthorsOpenReview ↗

Response to Reviewer e9BP (2/2)

W1 (continued). Real multi-page documents.

We also evaluate on MP-DocVQA (documents of 1-20 pages, about 512 tokens per page, Qwen2.5-VL-7B, full validation set of 5,187 questions), keeping 41 entries per layer for every request, so the retention ratio falls from about 8% for one page to below 1% for long documents.

Table R6. MP-DocVQA (Qwen2.5-VL-7B).

MP-DocVQA, 41 entries per layerAll1 page2-5 pages6-10 pages11-20 pages
Full cache78.887.379.172.069.1
TGV-KV55.370.654.745.937.7
H2O45.957.944.639.532.8
SnapKV34.535.634.834.631.6
CARVE72.280.872.868.759.1
CARVE-G69.879.970.763.156.3

CARVE keeps 92% of the full-cache ANLS against 70% for TGV-KV, with the largest gap (21-23 points) on documents of six pages or more.

W2. Hyperparameters.

All constants are fixed once and shared across the 64 model-benchmark-budget settings. The detection constants were set on the first 128 questions of the InternVL3-8B ChartQA test and DocVQA validation splits, and excluding these questions changes CARVE by at most 0.7 points (DocVQA 69.5 → 69.3, ChartQA 66.8 → 67.5 at 1%) and TGV-KV by at most 0.3, so every comparison is unchanged. The revision removes two constants (the 35-token threshold and the blend weight ω\omega of CARVE-G). Table R7 lists the remaining ones.

Table R7. Constants of CARVE.

ConstantValueModelSensitivity at 1%, DocVQA / ChartQA (chosen value in bold)
Cells RR128InternVL370.7 / 58.4 (R=32R=32), 70.6 / 65.8 (64), 69.5 / 66.8 (128), 64.0 / 63.7 (256)
Qwen2.577.4 / 60.8 (32), 77.6 / 61.6 (64), 78.4 / 60.2 (128), 76.8 / 54.4 (256)
Funded fraction0.2InternVL3DocVQA only, 66.3 / 69.5 / 70.4 / 69.8 for 0.1 / 0.2 / 0.3 / 0.4
Committed share ff0.75InternVL365.5 / 66.5 (0.5), 69.5 / 66.8 (0.75), 70.4 / 66.1 (0.9)
Qwen2.572.9 / 58.5 (0.5), 78.4 / 60.2 (0.75), 79.8 / 59.8 (0.9)
Prompt-length switchremovedall fourremoving it raises CARVE's 1% mean on every model (full sets, +0.4 to +1.8 points)
Micro-budget switch≤16 entriesall fourneeded only on LLaVA-NeXT (-5.2 without it), and removing it changes the other models by +0.5 to +0.9

(Full evaluation sets, prompt-length switch removed.) Accuracy changes little around the chosen values (R between 64 and 128, committed share between 0.75 and 0.9), which are not the per-benchmark maximizers, and the settings that cost more than two points lie at the ends of the tested ranges. The signal and gate thresholds act only on a minority of sample-layer pairs (signal switch 0.1-9.4%, closed gate 0.2-27.5%, our answer to Reviewer Dnaf, Q2). Removing the signal switch changes CARVE's 1% mean by -3.3 to +0.5 points, and adding the gate by -3.4 to 0.0 at 1% and by -0.1 to +10.3 at 10% (Table R8 in our answer to Reviewer F5Ze).

W3. "Contiguous region".

You are right that token-level contiguity is not guaranteed. The sentence you quote describes where the evidence lies in the image. What CARVE guarantees is contiguity at the level of cells, as Section 3.5 of the revision now states. On the region path (89% of sample-layer pairs on InternVL3-8B at 1%), three quarters of the per-layer budget goes to tokens inside a few funded cells (about 4 on InternVL3-8B and 8 on Qwen2.5-VL-7B). Each cell is a contiguous rectangle of the token grid, so the retained tokens fall in a few compact regions, although tokens inside a cell are chosen by relevance and need not be adjacent.

In practice the retained tokens are clustered, and the clustering matters (InternVL3-8B DocVQA at 1%, full validation set).

  • Clustering. Across all 28 layers, 88.3% of the visual tokens that CARVE retains have another retained token among their eight grid neighbours, against 3.8% for the same number of tokens placed at random.
  • Effect on accuracy. Keeping the visual budget and cell count fixed but shuffling which tokens belong to each cell, so that each funded cell becomes a scattered set, costs 10.7 ANLS points (Section 6.1, signal switch off).
  • Visualizations. Figure 4 of the revision shows randomly drawn, correctly answered examples on all four benchmarks.

W4. Multi-image inputs.

Each image's token grid is partitioned separately, with the RR cells split across the images in proportion to their token counts (Appendix B). On four-page DocVQA inputs at 41 entries per layer, CARVE keeps 77.5 ANLS where the token-level methods score 34.8-51.9 (Table R5), and on MP-DocVQA documents of 11-20 pages it scores 59.1 against 37.7 for TGV-KV (Table R6).

Thanks for the response. I hope all additions and experiments can be reflected in the final version. I will improve my rating. Good luck!

Thank you for your positive feedback and encouragement. We have incorporated the additional experiments and clarifications into the revised manuscript and will ensure they are retained in the final version.

Reviewer_DnafOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies KV-cache eviction for vision-language models at extreme budgets, focusing on retaining just 1–3% of the input-context KV cache. Compression takes place after prefill and before the first decoding step; KV entries for generated tokens are subsequently appended. CARVE partitions visual tokens into layout-aligned spatial cells, ranks them by text-grounded relevance, and reserves most of the budget for tokens in a small set of highly ranked cells. It also includes attention-based selection and conservative fallback rules. CARVE-G adds confidence gating and modifies in-region scoring when the region path is active. The authors evaluate four 7B–8B models on four single-image visual question-answering benchmarks at four budgets, with generation capped at 32 tokens. The results show substantial gains at extreme budgets, while a stylized theoretical analysis and ablations examine the roles of allocation, spatial grouping, and fallback mechanisms.

Critical strengths
  1. The gains at extreme budgets are substantial. At a 1% budget, CARVE's four-benchmark mean exceeds that of the strongest evaluated baseline for each of the four models, by approximately 3.4–12.1 points (Table 1). These gains across models are my main reason for leaning toward acceptance.

  2. The ablations support the use of spatial structure: substantial gains remain with attention switching disabled, while shuffling spatial groups with the visual budget and region count held fixed reduces performance (Table 2a, Section 6.1). The authors also report cases where concentrated allocation fails, helping clarify when the method works.

Critical weaknesses
  1. The allocation ablation changes several factors at once. CARVE selects the top-Bv tokens from the union of selected regions, while Uniform-2D/Random-2D select tokens using per-region quotas. Their region selection and capacity allocation rules therefore differ. Under the integerization rule in Appendix B, with 128 regions and Bv=30, Uniform-2D also assigns one reserved slot to each of the 30 lowest-index regions. It is thus unclear how much of the gain in Table 2a comes from query-ranked region selection and how much is affected by these implementation differences.

  2. The robustness of the routing rules needs further validation. In Table 3, increasing the post-image prompt-length threshold from 30 to 35 raises InternVL3's DocVQA score by 5.4 points at a 1% budget. It would be useful to check whether semantically equivalent questions of different lengths also cause substantial performance changes. It is also unclear whether development samples overlap with the final evaluation. Furthermore, CARVE-G adds both gating and in-region reranking, so Table 2b does not isolate the gate's contribution.

Questions for authors
  1. How do Uniform-2D's results change if equal remainders are resolved randomly? With the region count, token selection and fill rules, and final visual/text retention held fixed, what is the performance gap between random and query-ranked region selection?

  2. Which splits supplied the ChartQA/DocVQA development samples, and do these overlap with the samples used for final evaluation? How was the 35-token threshold chosen? Please also report route usage rates over sample–layer pairs.

  3. Could you compare performance changes in CARVE/CARVE-G and TGV-KV under semantically equivalent rewrites that cross the 35-token threshold? Separately, with in-region scoring fixed, how much does adding the gate alone improve performance?

  4. How does SnapKV truncate its observation window and protected prefix when the budget is below 64 tokens? Please report the actual number of KV entries retained by each method in representative 1% settings. For the efficiency results, please specify the experimental configuration and compare end-to-end latency, including routing overhead, with the full-cache configuration.

3 comments
AuthorsOpenReview ↗

Response to Reviewer Dnaf (1/2)

We thank you for the detailed and insightful review. Two of your points led to concrete improvements. Your reading of Uniform-2D made us correct its tie-breaking, and your question on prompt length showed the prompt-length switch to be unnecessary, so we removed it.

W1 / Q1. Allocation ablation.

We are grateful for this catch. You are right that Uniform-2D breaks equal remainders by cell index. At 1%, BvB_v is smaller than the 128 cells (e.g., Bv=30B_v = 30 for a 4K-token prompt), so it funds the BvB_v lowest-index cells. We re-ran it with random tie-breaking. We also added the control you asked for, in which CARVE's own pipeline draws the funded cells at random instead of by the query ranking and everything else is unchanged (number of funded cells, token selection and fill rules, total and visual budgets).

Table R1. Allocation ablation.

1% budget, four-benchmark meanInternVL3-8BQwen2.5-VL-7B
Token-level top-kk (TGV-KV)41.052.7
Uniform-2D, ties to the lowest index (as submitted)42.452.8
Uniform-2D, random tie-breaking48.155.5
Random-2D47.655.5
Random funded cells, CARVE pipeline otherwise unchanged45.452.2
Query-ranked cells, per-cell quotas60.968.1
Query-ranked cells, CARVE (signal switch off)61.467.5
CARVE (signal switch on)64.766.9

(Full sets, prompt-length switch removed in all rows, unlike the submitted Table 2a.)

The gain comes mainly from letting the query choose which cells to fund. Drawing the funded cells at random costs 16.0 points on InternVL3-8B and 15.3 on Qwen2.5-VL-7B, whereas the capacity rule (union top-BvB_v versus per-cell quotas) changes the mean by at most 0.6. Correcting the tie-breaking raises Uniform-2D by 5.7 and 2.7 points, and the query-ranked allocation stays 12-13 points ahead. The signal switch adds 3.3 points on InternVL3-8B, almost all on ChartQA (53.0 → 66.8), and changes Qwen2.5-VL-7B by -0.5.

In the revision, Table 2a uses random tie-breaking, and Sections 3.4 and 6.1 no longer state that blind concentration outperforms blind spreading. Random allocations do not beat uniform spreading, as the uninformative endpoint of Proposition 1 predicts. Section 6.2 and Figure 3 are recomputed with this configuration (signal switch off in both), and the query-ranked allocation stays ahead of Uniform-2D in every entropy quintile of the pooled, 1% and DocVQA panels.

W2 / Q2. Routing rules, development samples, and the 35-token threshold.

  • Development samples. The detection constants were set on the first 128 questions of the InternVL3-8B ChartQA test and DocVQA validation splits, so these questions are part of the evaluation sets. Excluding them changes CARVE by at most 0.7 points (DocVQA 69.5 → 69.3, ChartQA 66.8 → 67.5 at 1%) and TGV-KV by at most 0.3, so every comparison is unchanged.
  • The threshold. The 35-token switch was set during development as a conservative guard for long prompts. Question length, however, says nothing about where the visual evidence lies. On InternVL3-8B at 1%, DocVQA questions of 36-40 tokens score 35.1 under the switch and 69.1 when routed, and equivalent rewrites that cross the threshold revert the sample to the base selection (Q3). Its benefit is confined to the ungated mode at 5-10%, mainly on ChartQA, which the gate of CARVE-G covers. We therefore remove the switch from both modes. At 1% this raises CARVE's four-benchmark mean on all four models (+0.4 to +1.8 points, e.g., 62.9 → 64.7 on InternVL3-8B). Under rewrites, CARVE now changes by at most 2.4 points (Q3).
  • Route usage over sample-layer pairs. The region path makes most decisions on InternVL3-8B and Qwen2.5-VL-7B, the two models with the largest 1% gains over the best baseline (+12.3 and +13.5 points). On LLaVA-NeXT and Qwen3-VL-8B, many layers keep at most 16 entries at 1%, so about 42% of pairs return to the base selection through the micro-budget fallback, and the 1% gains are smaller (+4.3 and +3.8). This fallback still keeps LLaVA-NeXT from losing 5.2 points. CARVE-G closes its gate most often on InternVL3-8B, where adding the gate costs 3.4 points at 1% but gains 10.3 at 10% (Q3). No pair is reverted by prompt length any more (Table R9).

Table R9. Route usage at 1% (% of sample-layer pairs, revised CARVE, full sets, four benchmarks pooled).

RouteInternVL3-8BLLaVA-NeXTQwen2.5-VL-7BQwen3-VL-8B
Region path89.354.670.234.3
Signal switch4.13.79.40.1
Micro-budget fallback3.541.74.341.9
Filled by protected entries (short inputs)3.00.016.223.7
Gate closed (CARVE-G)27.53.710.10.2
AuthorsOpenReview ↗

Response to Reviewer Dnaf (2/2)

Q3. Rewrites and the gate.

  • Rewrites. We rewrote the DocVQA and ChartQA questions into equivalent forms that are about 13 tokens longer and cross 35 tokens, using Qwen2.5-7B-Instruct with an equivalence check. A valid rewrite exists for 4,508 DocVQA and 1,939 ChartQA questions, and we keep those whose full-cache answer is unchanged (3,671 and 1,512).

Table R2. Original → rewritten questions (Qwen2.5-VL-7B, 1%).

DocVQA original → rewriteChartQA original → rewrite
Full cache95.4 → 95.485.5 → 85.5
TGV-KV64.5 → 43.255.3 → 54.9
CARVE with the prompt-length switch82.0 → 47.862.6 → 56.1
CARVE (switch removed)82.0 → 79.662.5 → 64.7
CARVE-G (revised)81.8 → 79.161.6 → 63.1

As you anticipated, with the switch, rewrites that cross the threshold revert CARVE's region routing to the base selection. Without it, CARVE stays within 2.4 points on Qwen2.5-VL-7B, while the base scorer itself drops from 64.5 to 43.2 on DocVQA, and on InternVL3-8B it goes from 72.6 to 73.6 on DocVQA (72.6 → 36.2 with the switch). Over all rewrites, including those that change the full-cache answer, CARVE changes no more than the full cache itself (-4.2 and +0.4 against -4.6 and -5.9 on Qwen2.5-VL-7B DocVQA and ChartQA).

  • Gate alone. With in-region scoring fixed to the base relevance, adding the gate to CARVE changes the four-benchmark mean by -3.4 / -0.5 / -0.5 / 0.0 at 1% and +10.3 / 0.0 / -0.1 / +0.2 at 10% (InternVL3-8B / LLaVA-NeXT / Qwen2.5-VL-7B / Qwen3-VL-8B, full sets), and on InternVL3-8B ChartQA at 10% it restores 82.9 from 49.1. The gate thus accounts for CARVE-G's behavior, and we simplify CARVE-G to CARVE plus the gate, with the ungated CARVE remaining the mode for 1-3%. Together with the removal of the prompt-length switch, this changes CARVE-G's four-benchmark mean by -0.1 to +1.7 points relative to the submitted version (general response).

Q4. SnapKV, retained entries, efficiency.

  • SnapKV. SnapKV ranks its 64-entry observation window above all other entries. When the per-layer budget is below 64, the retained entries all come from that window (the most recent prompt positions) plus the protected first and last entries, so its selection reduces to recency. At 1%, SnapKV is identical to StreamingLLM on InternVL3-8B and LLaVA-NeXT and within 0.5 points of it on the Qwen models. It keeps no more entries than any other method.
  • Retained entries. Every method keeps the same number of context entries in every layer, B=⌈0.01N⌉B=\lceil 0.01N\rceil for a prompt of NN tokens (measured retention 0.0100). In representative 1% settings this is about 18 entries per layer for InternVL3-8B on DocVQA (prompts of about 1.8K tokens), 24 for LLaVA-NeXT (2.4K) and 41 for Qwen2.5-VL-7B (4.1K). Generated tokens (≤32) are appended uncompressed for all methods.
  • Efficiency. Batch size 1, greedy decoding, eager attention, one RTX 4080 SUPER (32 GB variant). Prefill is unchanged. Relative to TGV-KV, the routing adds 30 ms per sample for CARVE and 60 ms for CARVE-G on InternVL3-8B at 1% (7% and 14% of the prefill forward pass), and removing the prompt-length switch leaves this cost unchanged. Decoding depends only on the budget, which is the same for every method, and the saving is the memory held during decoding. Table R3 gives the decode throughput with the 1% cache on the same GPU (language model of InternVL3-8B, 32 greedy decoding steps).

Table R3. Decode throughput (tok/s), full cache → 1%.

Visual contextBatch 8163264
8 images (34K tokens)172 → 327 (1.9$\times$)OOM → 638 (3.5$\times^†$)OOM → 1,136 (6.3׆\times^†)OOM → 2,119 (11.7\times^†\)

†^†The full cache fits at most 9 eight-image requests, so the ratio is taken against that batch (181 tok/s). For a single high-resolution page, the GPU fits 73 concurrent requests with the full cache against more than 1,024 at 1%, with 4.7$\times$ the decode throughput at that concurrency (Table R4 in our answer to Reviewer e9BP).

AuthorsOpenReview ↗

Response to Reviewer Dnaf - Closing Note

As the discussion period comes to a close, we thank you again for your review.

Your two catches changed the paper. Uniform-2D now breaks ties at random, and the new control that funds random cells within CARVE's otherwise unchanged pipeline isolates the contribution of query-ranked region selection. Your question on prompt length led us to remove the prompt-length switch, so the routing no longer switches on a question-length threshold.

We hope our response also answered your questions on the development samples, the gate and the retained entries.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response to all reviewers and area chairs

We sincerely thank Area Chairs and all three reviewers. Their reviews led us to simplify the method (removing a prompt-length rule that made the selection depend on question length), to correct the tie-breaking of one ablation baseline, to state the contiguity claim precisely, and to make the motivation concrete. All new results below use the full evaluation sets and are included in the revised PDF.

1. A simpler method (Reviewers Dnaf and F5Ze)

We drop two rules, and CARVE becomes simpler without losing accuracy.

  • We remove the prompt-length switch (35-token threshold) from both modes, and CARVE-G becomes CARVE plus the confidence gate (the blend weight ω\omega is dropped). No other constant changes.
  • Question length says nothing about where the evidence lies, and the gate-alone ablation shows that the protection of CARVE-G comes from the gate (Dnaf Q3).
  • At 1%, the four-benchmark mean of CARVE rises on every model (62.9 → 64.7 on InternVL3-8B, 37.5 → 38.1 on LLaVA-NeXT, 65.6 → 66.9 on Qwen2.5-VL-7B, 64.7 → 65.1 on Qwen3-VL-8B).
  • CARVE-G changes by -0.1 to +1.7 points across the sixteen model-budget settings and still stays within 1.6 points of its fallback on every benchmark, as in the paper.

2. Why 1-3% (Reviewers e9BP)

1-3% is where long visual inputs land when the memory per request is fixed, and at these ratios the eviction policy decides how much of the full-cache accuracy survives.

  • On one 32 GB GPU, an eight-image context admits 9 concurrent requests with the full cache, 92 at 10%, and 840 at 1% (Table R4 in our response to Reviewer e9BP).
  • With 41 entries per layer, CARVE keeps 92% of the full-cache ANLS on real documents of 1-20 pages (MP-DocVQA), where TGV-KV keeps 70% (Table R6 in the same response).

3. Cleaner allocation controls (Reviewer Dnaf)

The gain comes mainly from letting the question choose which regions to fund, not from partitioning the image into cells.

  • We keep CARVE unchanged except that the funded cells are drawn at random instead of ranked by the question. At 1%, the four-benchmark mean drops by 16.0 / 9.4 / 15.3 / 7.5 points (InternVL3-8B / LLaVA-NeXT / Qwen2.5-VL-7B / Qwen3-VL-8B, our answer to Reviewer Dnaf, W1).
  • Uniform-2D with random tie-breaking stays 12-13 points below the query-ranked allocation.

4. Further analyses

Each remaining component has a measured role, and CARVE is robust to its constants and to how the question is phrased.

  • Component ablations on all four models, including the gate alone (Reviewers Dnaf and F5Ze).
  • Sensitivity to the cell count RR and the committed share ff (Reviewers e9BP and F5Ze).
  • Robustness to rewritten questions and prompt templates, and efficiency measurements (Reviewers Dnaf and F5Ze).
  • Visualizations of the retained tokens (Reviewer e9BP).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.