- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper studies query-agnostic visual KV-cache compression, where an image cache must be compressed before downstream questions are available. Synthetic Question Attention (SQA) supplies spatial relevance scores, and MEKV interpolates between spatial coverage and relevance-based allocation using a temperature-controlled exponential family. Tokens are sampled uniformly within the resulting within the resulting regional quotas. The paper establishes concavity of an oracle-attention KL surrogate along this family and uses concentration diagnostics to interpret differences between Qwen and InternVL. Experiments report improvements over coverage baselines on document, chart, and scene-text VQA, together with a calibrated per-head selection extension.
Critical strengths
1. The empirical observation about excessive concentration is useful. Informative proxy-attention scores can produce worse retention decisions than spatial coverage when used too aggressively. The Qwen results provide a concrete example of this failure, motivating calibration as a relevant consideration for reusable visual caches.
2. Some improvements are supported by substantial evaluation. The fixed-temperature InternVL experiments cover full validation sets across three benchmarks and report paired bootstrap intervals. The calibrated per-head extension also includes a DocVQA evaluation explicitly disjoint from development data. These results provide evidence that balancing relevance and coverage coverage can improve particular compression policies.
Critical weaknesses
1. The theoretical results do not establish the central claims about task accuracy or architecture dependent strategy selection. Theorem 1 establishes strict concavity of , which follows directly from the exponential-family construction. However, is an attention distribution obtained from real questions, and the paper provides no bound or consistency argument connecting this surrogate to ANLS or VQA accuracy after discrete, capacity-constrained token retention. Matching an attention moment therefore does not establish that the resulting cache preserves the evidence required to answer unseen questions. The extension to the "architecture reversal" is particularly under-supported. Proposition 1 decomposes allocation concentration, not prediction error. Its nonnegative off-path term does not determine whether a policy's additional structure improves accuracy. Likewise, a local surrogate expansion along the temperature family does not establish an accuracy ordering between MEKV and per-head top-, which generally lies outside that family. The quantitative diagnostic is also inaccurate in important reported settings. At 20% retention, the moment-matching estimates are approximately 0.613 versus a measured grid optimum of 0.25 for Qwen, and 0.996 versus 0.42 for InternVL, as shown in Appendix C. These are discrepancies of roughly a factor of 2.4. Corollary 2 additionally assumes that tightening the budget increases the oracle moment, but the operational oracle in Appendix C.1 is defined from real-question attention without specifying how it becomes budget-dependent. Figures 4 - 6 consequently provide descriptive observations rather than strong validation of the proposed theoretical explanation. In particular, failing to detect a statistically significant secondary peak does not establish unimodality. This gap matters because the explanatory framework is presented as a principal contribution beyond temperature tuning.
2. The experiments do not sufficiently isolate the proposed contribution or establish its advantage under a consistent eavluation protocol. The headline gains of up to 17 percentage points are against coverage baselines. The stronger same-signal comparison gives a substantially different picture: on InternVL DocVQA, MEKV scores 75.56/56.40, while SnapKV-style selection scores 77.89/64.28 at 20%/10% retention. SnapKV-style selection also outperforms MEKV on InternVL textVQA, shown in Table 13. Against optimized linear signal-coverage mixing, the reported temperature advantage is only 1.13/2.03 points on Qwen and is statistically unresolved on InternVL, shown in Appendix H. Those comparisons use the best settings from sweeps and do not establish held-out superiority of the exponential parameterization. Evaluation coverage and splits are fragmented. Table 13 explicitly uses development evaluation for Qwen DocVQA but full validation for InternVL and the TextVQA comparisons. Appendix B does not establish that the full DocVQA validation results exclude temperature-selection images. Appendix I provides a disjoint evaluation for the hybrid extension, but this does not resolve the corresponding question for the base method. Furthermore, there is no consolidated comparison containing uncompressed performance and all stated baselines across the principal model–benchmark settings. Without uncompressed scores, the accuracy cost associated with the reported cache savings remains difficult to assess. The hybrid extension also changes two factors simultaneously: it introduces regional quotas and replaces global deterministic top- with tempered sampling within regions. Consequently, Table 2 does not isolate how much improvement comes from regional allocation, sampling, or their interaction. This is consequential because the experiment is used to support the paper’s causal interpretation of concentration control. Finally, the third-architecture evidence has a major scope limitation. In Gemma’s primary “full8” arm, the layers being compressed account for only 4.76% of total visual KV capacity, while the remaining layers retain their visual KV (Appendix B.4). The nominal 20%/10% retention settings therefore correspond to approximately 3.8%/4.3% reductions in total visual KV capacity under that accounting. This is a much narrower compression setting than the headline retention percentages suggest.
Questions for authors
1. Evaluation clarification. Where are the DocVQA development examples drawn from, does "DocVQA-300" count images or questions, and which reported evaluations exclude those images? Please identify the splits and score aggregation conventions for the base method, baselines, and hybrid extension.
2. Evidence needed to assess the empirical contribution. Can you provide a common, image-disjoint comparison of base MEKV, calibrated per-head selection, tuned linear mixing, the stated compression baselines, and the uncompressed model? Please specify the baseline adaption parameters. For example, the filtering layer is an explicit parameter of the original FastV method, but its value is not specified in Appendix B.3.
3. Clarification of the theoretical claims. What justifies using oracle-attention FL as a surrogate for downstream accuracy? How is the budget-dependent oracle in Corollary 2 constructed, and which result supports predicting the relative accuracy of an off-path top- policy?
4. Evidence needed for the causal interpretation. What happens when regional quotas and the within-region selection rule are varied independently? In particular, do calibrated quotas still repair Qwen when selection within each region remains deterministic top-, and how much improvement is obtained from tempered sampling without regional quotas?
3 comments
Official Comment of Submission54483 by Reviewer Q9A7
Thank you for the detailed responses. The revisions and additional experiments have resolved nearly all of my concerns, and I have increased my rating. My remaining concern is the lack of a comparison between MEKV and linear signal-coverage mixing on the same held-out images, with both methods' parameters selected on the development set and then frozen. This would clarify whether the exponential-family parameterization offers an advantage beyond the general benefit of balancing relevance and coverage, since the current best-of-sweep development results do not establish held-out superiority.
Dear Reviewer Q9A7,
Thank you for your careful re-evaluation and for raising your rating. Your detailed feedback led to substantial improvements in our revision.
Regarding your remaining point, we have launched the held-out comparison with linear signal-coverage mixing on DocVQA-786 and full ChartQA, with selected per budget solely on DocVQA-300 and frozen across benchmarks, under the matched 3-seed protocol. The multi-seed runs are still in progress, and we will include the full comparison in Appendix H of the final manuscript.
Best regards,
The Authors
Thank you for the thorough review; it drove the main structural, experimental, and scoping revisions in the updated PDF.
Q1. Splits and aggregation (§5, Appendix B.3). DocVQA validation (1,286 images) is split into three disjoint subsets: DocVQA-300 (300 images / 1,799 questions), used solely to select fixed parameters; 200 images from earlier method checks, excluded from both selection and testing; and strictly held-out DocVQA-786. ChartQA and TextVQA use full validation sets, fully held out. All main Qwen results are now on held-out splits (Table 2). Excluding DocVQA-300 from InternVL full validation (DocVQA-986) leaves every conclusion unchanged: Base MEKV − Stratified is +5.2 [4.1, 6.3] / +7.6 [6.3, 8.9] pp at 20% / 10%; SnapKV-style − Base MEKV is +2.6 [1.3, 3.9] / +8.6 [7.1, 10.2] pp. All scores are image-mean ×100 (questions averaged within each image, then across images).
Q2. Consolidated comparison and baseline parameters (Table 2; Appendices B.3, G, H). Image-mean ×100, 20% / 10%:
† pooled 3-seed per-head harness matched to Calib. Per-Head; ‡ held-out DocVQA-786.
Q3. Scope of theoretical claims (§4, §5.2, Appendix C.5). We agree and have narrowed the claims. We do not claim a bound from −KL(a∥p(λ)) to ANLS; the surrogate is an analytical model, and λ^ is a regime diagnostic and search initializer (it over-concentrates at generous budgets in direct no-search runs and converges at 5%; Table 7). Corollary 2 is now stated as conditional; since our oracle has no budget parameter, the budget shift in Figure 5 is reported as an empirical regularity. Proposition 1 decomposes concentration, not error, and does not rank off-path policies; direct evidence for the reversal comes from the ablations in §5.3. The absence of a significant secondary peak is described as consistent with single-peakedness, not proof of it.
Q4. Quotas and within-region selection varied independently (Table 3; Appendix I.1). On held-out DocVQA-786 and full ChartQA (full table in our response to Reviewer zzSY): on Qwen, calibrated quotas with deterministic within-region top-k repair the failure in all four settings (+4.2 / +5.0 pp DocVQA, +4.2 / +8.5 pp ChartQA, all significant); tempering without quotas adds +6.7 to +11.8 pp; adding quotas atop tempering adds a further significant +2.1 to +4.4 pp in three of four settings. On InternVL, quotas alone bring no significant change, while tempering helps (+0.9 to +4.4 pp), consistent with its near-raw regime.
Gemma scope. We agree with your accounting: full8 compresses only the 8 full-attention layers (4.76% of visual KV, a 3.8–4.3% whole-cache reduction). The revision frames it as a full-attention-layer probe in a hybrid architecture, alongside the all48 arm (Table 10).