- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
CARVE partitions visual tokens into layout-aligned cells, ranks cells using query-conditioned token relevance, and reserves most of the cache budget for tokens within selected cells. Selection-time diagnostics can switch to attention-based selection; CARVE-G adds a confidence gate that falls back to the underlying token-level method. Compression occurs after prefill and before decoding, with a per-layer context budget of .
The paper evaluates four VLMs on four visual question-answering benchmarks at cache retention ratios of , , , and . On InternVL3-8B DocVQA at retention, CARVE achieves 66.8 ANLS, compared with 35.5 for TGV-KV and 89.8 for the full cache. A stylized retrieval analysis motivates concentrating the budget when region rankings are reliable and the budget is small.
Critical strengths
- Substantial improvements under a controlled core comparison. CARVE uses the same base relevance signal as TGV-KV and the same total cache budget. The large DocVQA gains therefore provide meaningful evidence for changing the selection policy rather than merely improving token scoring (pp. 5–7, Table 1).
- Ablations directly probe allocation and geometry. The comparison with one-dimensional selection, uniform allocation, and random allocation is complemented by an experiment that shuffles cell membership while preserving the budget and cell count. The resulting degradation supports the importance of spatially coherent groups (pp. 7–8, Table 2a and Section 6.1).
- Broad evaluation within the stated task family. Four models, four benchmarks, and four budgets provide a reasonably informative picture of the method's operating regimes. The paper reports cases where other methods remain stronger.
- The gated variant addresses a real failure mode. CARVE can regress at larger budgets, and CARVE-G substantially reduces these regressions. Its worst single-benchmark difference relative to TGV-KV is reported as points, rather than being presented as a universal improvement (Tables 1–2).
Critical weaknesses
- The full method relies on several insufficiently isolated heuristics. These include the attention/text signal switch, a micro-budget fallback, a 35-token post-image prompt threshold, multiple confidence thresholds, and hybrid scoring. The current ablations do not fully establish the necessity and portability of each rule (pp. 16–18, Appendix B).
- The theoretical result does not directly characterize the implemented algorithm. The analysis assumes a single evidence region and a concave retrieval utility, with its detailed reliability characterization restricted to two regions. The implementation instead uses max-pooled scores, discrete region selection, and fixed heuristics. The paper acknowledges that entropy, Gini, and top-mass statistics are not formally linked to ranking correctness (pp. 14–16).
- Context-cache retention is not total-memory reduction. The model still processes the full input during prefill, and newly generated tokens are appended afterward. Evaluation limits generation to 32 tokens. Thus, retained input-context entries should not be interpreted as total GPU memory or a demonstrated long-generation operating point (pp. 3 and 9).
- The practical efficiency evidence is incomplete. Decode latency is similar across methods, while CARVE and CARVE-G add approximately 30 ms and 60 ms of routing overhead per sample on InternVL3. Concurrent serving throughput, peak memory, and realized request capacity are not measured. Evaluation also remains single-image and short-answer despite the broader long-document motivation (p. 9).
Questions for authors
- Can you report a systematic ablation of the prompt-length fallback, micro-budget fallback, signal switch, and confidence gate across all four models? Which components account for most of the improvement?
- How were the development probes used to choose the routing thresholds separated from the final evaluation examples? How well do these thresholds transfer to longer questions and different prompt templates?
- Can you report actual KV-cache bytes, end-to-end latency including prefill and routing, and concurrent serving throughput at matched answer quality?
- How sensitive is the method to cell granularity and to questions requiring evidence from several spatially separated regions? Does the advantage persist with longer generated answers?
4 comments
Response to Reviewer F5Ze (2/2)
Q2. Development probes and transfer.
The detection constants were set on the first 128 questions of the InternVL3-8B ChartQA test and DocVQA validation splits, so these questions are part of the evaluation sets. Excluding these questions changes CARVE by at most 0.7 points (DocVQA 69.5 → 69.3, ChartQA 66.8 → 67.5 at 1%) and TGV-KV by at most 0.3, so every comparison is unchanged. CARVE also transfers to longer questions and other templates. With a longer instruction (+15 tokens), it goes from 69.5 to 70.0 ANLS on InternVL3-8B DocVQA at 1% and from 78.4 to 77.6 on Qwen2.5-VL-7B, where the full cache itself drops by 2.2 points. With an alternative phrasing of the same length, it goes from 69.5 to 70.6 on InternVL3-8B DocVQA, and on ChartQA either template raises it from 66.8 to 69.5. The prompt-length switch we removed would instead have reverted every prompt of the longer template to the base selection (34.4 and 39.7 ANLS).
Q4. Cell granularity, multi-region evidence, longer answers.
- Granularity. Accuracy stays within 1.5 points for R between 64 and 128 cells and drops for very fine cells (1.6 to 5.8 points lower at on DocVQA and ChartQA), where a funded cell holds too few tokens to cover an answer region.
- Multi-region evidence. InfoVQA annotates answer types and operations. At 1% on InternVL3-8B, CARVE improves most on single-span answers (44.1 → 59.3 ANLS over TGV-KV), less on multi-span answers (20.7 → 22.2) and on questions requiring comparison, arithmetic or counting (46.9 → 51.9). Multi-span answers remain hard for every method at 1%.
- Longer answers. Our benchmarks follow the short-answer protocol (≤32 generated tokens), so we make no claim about long generations. The retained context is fixed after prefill and generated tokens are appended for every method, so a longer answer draws on the same retained evidence.
Thank you for the detailed response and additional experiments. The rebuttal strengthens the paper and addresses several of my concerns.
The corrected allocation controls are particularly helpful. Query-ranked allocation retains a substantial advantage after correcting Uniform-2D’s tie-breaking, and replacing query-ranked cells with randomly selected cells within the otherwise unchanged CARVE pipeline causes large performance drops. This provides stronger evidence that query-conditioned region selection drives the gains. I also appreciate the removal of the prompt-length switch and hybrid scoring, together with the component ablations and question-rewrite experiments. The new MP-DocVQA results and the four-page comparison at near-matched quality make the practical motivation more convincing.
The response has sufficiently addressed my questions for this discussion. My overall assessment remains that CARVE makes a useful empirical contribution through query-conditioned region allocation under extreme post-prefill cache budgets. The revised controls and simplified routing policy provide a clearer account of why it works. I regard the theoretical analysis as useful motivation, and assess the implemented method on the strength of its empirical evidence.
Taking the contribution and its scope together, I remain supportive of acceptance at the level of weak acceptance and will maintain my rating of 3. The additional results strengthen this recommendation. My rating reflects my overall assessment of the paper’s contribution, with the strongest evidence concerning short-answer visual QA under severe context-cache constraints. I have no further questions at this stage.
Response to Reviewer F5Ze - Thank You
We thank you for the careful follow-up and for maintaining your support. Your request to justify each rule led to the systematic ablation and to the removal of the prompt-length switch, and your questions on serving capacity and multi-page inputs made the case for the 1-3% regime concrete. We agree with your reading of the scope: the evidence is strongest for short-answer visual QA under severe context-cache budgets, which is the scope our Limitations paragraph states.
Response to Reviewer F5Ze (1/2)
We thank you for the constructive assessment. Your request to establish the necessity of each rule was especially valuable. The systematic ablation it prompted showed that the prompt-length switch contributed nothing at 1-3% and made the selection depend on question length, so we removed it, and it clarified what the remaining rules do.
W1 / Q1. Heuristics.
We now report a systematic ablation on all four models (Table R8). Each entry is the change in the four-benchmark mean when one component is removed from (or, for the gate, added to) CARVE without the prompt-length switch, and the two prompt-length rows compare the submitted CARVE with and without the switch (full sets).
Table R8. Component ablation.
The region allocation carries the gain. The signal switch matters on InternVL3-8B ChartQA (53.0 → 66.8). The micro-budget switch matters when the per-layer budget falls to about a dozen entries (LLaVA-NeXT at 1%), and we keep it because it prevents a 5.2-point loss there while costing at most 0.9 points elsewhere. The gate protects the loose budgets (49.1 → 82.9 on InternVL3-8B ChartQA at 10%), while CARVE is the mode for 1-3%.
W2. Theory.
The analysis is indeed stylized. It motivates concentration and the use of reliability diagnostics, and does not characterize the implemented selection. The revision states its assumptions next to the result (Appendix A) and presents it as motivation. Its uninformative endpoint is also visible in the corrected ablation, where random allocations do not beat uniform spreading at 1% (Uniform-2D 48.1 / 55.5, Random-2D 47.6 / 55.5 and random funded cells in CARVE 45.4 / 52.2 on InternVL3-8B / Qwen2.5-VL-7B in Table R1 of our answer to Reviewer Dnaf).
W3. Context-cache retention versus total memory.
The revision makes the wording precise. A ratio of 1% refers to the context cache retained after prefill, while prefill processes the full input and generated tokens are appended. It reports bytes explicitly (Table R4 below and Table 7 of the revision) and makes no claim about total or peak memory.
W4 / Q3. Efficiency.
Table R4 gives the KV bytes, the measured concurrency each ratio admits and the resulting decode throughput.
Table R4. Context-cache size and serving capacity (copied from our response to Reviewer e9BP for your convenience).
(Measured on one 32 GB GPU with the language model of InternVL3-8B, which leaves about 17 GiB for the context cache after the weights, with 32 greedy decoding steps and batch size searched up to 1,024.)
Going from 3% to 1% raises decode throughput by 1.5× for an eight-image context and 2.3× for a 131K-token context. This serving gain holds for any eviction policy, and CARVE changes the accuracy at that budget, including on multi-page inputs. With 41 entries per layer, it keeps 92% of the full-cache ANLS on MP-DocVQA against 70% for TGV-KV (Table R6 in our answer to Reviewer e9BP). At near-matched quality (within 0.7 points), CARVE needs a third of the cache, scoring 77.5 on four pages with 41 entries per layer against 78.2 for TGV-KV with 123 (Table R5 in the same answer).
Latency. Prefill processes the full input for every method. On InternVL3-8B at 1% with batch size 1, the routing adds 30 ms per sample for CARVE and 60 ms for CARVE-G (7% and 14% of the prefill forward pass), and every method decodes at 25-26 ms/token (Section 6.5 of the revision and our answer to Reviewer Dnaf, Q4).