- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates whether similarity-based acceptance criteria for cross-layer KV-cache sharing can reliably preserve visual capabilities in VLMs. The authors demonstrate that conventional similarity metrics and aggregate VQA scores may fail to capture degradation in object–attribute binding. To address this, they introduce RecallGate, which evaluates each proposed sharing operation using a small capability-specific probe set under the cumulative compressed configuration. Experiments across multiple VLM families show improved binding retention in several matched-compression comparisons, while revealing sensitivity to probe selection and interactions between shared layers.
Critical strengths
The paper highlights an important limitation of conventional compression evaluation: representation similarity and aggregate task performance may not reflect the preservation of specific visual capabilities. The investigation of object–attribute binding provides a useful perspective on compression-induced degradation.
Critical weaknesses
1.The evaluated Q Cache, MiniCache-inspired, and KVSharer-inspired implementations are reconstructed or adapted rather than faithful reproductions. Notably, the Q Cache reconstruction fails to recover the original operating point and exhibits severe degradation in aggregate task performance. Consequently, the conclusions may reflect properties of the adapted implementations rather than limitations of the original methods.
2.RecallGate uses only 16 synthetic binding examples by default, which may provide an unstable acceptance signal. Repeatedly evaluating candidates on the same battery also introduces potential adaptive overfitting. The observed sensitivity to battery size, acceptance thresholds, and random seeds raises concerns about the reliability of the proposed acceptance decisions.
3.The controlled SmolVLM2 experiments reveal substantial performance variation between configurations selected under identical compression budgets. In particular, RecallGate configurations A and B achieve 89.7% and 58.3% layout binding accuracy, respectively. Despite evaluating cumulative configurations, RecallGate can still accept combinations that exhibit severe joint degradation. This limits the robustness of the proposed selection mechanism.
4.Several comparisons rely on limited evaluation samples or fixed selected configurations. Although the authors report confidence intervals for certain comparisons, the controlled selection experiment contains only five runs producing two distinct configurations. Stronger selection-level statistical analyses across additional models and independent probe distributions would improve the reliability of the conclusions.
Questions for authors
1.What is the fundamental methodological contribution of RecallGate beyond replacing conventional similarity-based acceptance with capability-specific validation?
2.How can the authors ensure that the reported findings generalize to the original KV-cache compression methods, given that several evaluated implementations are reconstructed or adapted?
Response to xXuq: Contribution, Reconstruction Scope and Selection Stability
We thank the reviewer for the positive assessment and for recognizing that representation similarity and aggregate scores can miss capability loss. We ran new experiments on battery stability, real-image transfer, newer models and larger evaluations. Accuracy-difference CIs are paired 95% bootstrap intervals resampling whole images; differences are in percentage points.
Q1. Contribution beyond capability-specific validation. RecallGate is deliberately simple; the contribution is what it reveals and controls. (i) Audit of two tested acceptance criteria: Q Cache's static attention-divergence score has near-zero correlation with retained binding across 104 single shares in four families (family-stratified ρ = +0.009, descriptive), it admitted the most destructive share available in two families, and POPE and GQA stay near baseline while binding collapses; a cumulative hidden-state cosine test, as in KVSharer, admits Qwen2-VL MC-R configurations spanning 73% to 0% binding at one threshold. (ii) Two distinctions the field conflates: single-substitution vs cumulative evaluation, and representation similarity vs capability. The second matters even when the first is handled: cumulative installation with a similarity test still admits severe binding loss. (iii) Controlled evidence that adding a capability check improves selection: at the same threshold, candidate order and number of shared layers, RG keeps more binding at 17/17 matched doses where selections differ (+8 to +69 points; table in our response to juBr). (iv) Selectivity: a spatial-relation battery improves held-out spatial accuracy by +7.0 [+1.3, +13.0] over the proxy but does not protect binding, supporting capability-specific validation, without establishing preservation of either tested or untested capabilities (response to bMQf, Q1).
W1/Q2. Reconstructions and generalization to the original methods. We agree this bounds our claims, and we state it as scope. The audited object is the acceptance decision: our rebuild of Q Cache's criterion reproduces its reported acceptance volumes and is validated component by component, while its published operating point did not reproduce. The 104-share audit evaluates single shares on unmodified models, so it does not depend on that operating point. KS-R and MC-R borrow ranking principles from KVSharer and MiniCache and use a cumulative hidden-state cosine test; we do not evaluate their complete implementations. The results extend across model families and newer checkpoints under our implementation: on Qwen3-VL-2B and InternVL3.5-1B, proxy approval collapses binding at only two shared layers (79.0 → 17.0, 98.3 → 58.3) while RG keeps 77.0 and 90.3. They do not establish that the original methods exhibit the same failures; the revision states this.
W2/W4. Battery size, adaptive overfitting and selection-level statistics. We added selection-level repeats with independent gate seeds:
A larger battery stabilizes selection in the tested setting, and the newer models are mostly stable; the Qwen2-VL spread remains and is reported as is. Gate seeds measure selection sensitivity; transfer to evaluations the gate never saw provides generalization evidence. On real-photo GQA colour questions (n=400 on 392 images), RG beats the proxy in all five matched cells, with intervals excluding zero in three (e.g. +7.2 [+3.8, +10.9]); on 491 POPE items, after excluding items whose images were used for calibration, RG wins 4, the interval includes zero in 1, and RG loses 1 (OneVision KS-R). Neither rules out adaptation to the repeatedly reused battery, which remains a risk; RG is a comparative improvement, not a guarantee.
W3. Configurations A and B. Both A and B outperform proxy approval on the original binding evaluation at two shared layers (85.7% and 77.3% versus 23.0%). However, B's 58.3% layout accuracy remains a substantive generalization failure. Enlarging the battery to 32 items selects A in all five tested seeds; this reduces selection of B in that setting, without establishing universal robustness.
We thank the reviewer again, and hope these results strengthen confidence in the paper's conclusions.