acceptodds
Under review as a conference paper at ICLR 2027

Auditing Capability Retention under Cross-Layer KV Sharing in Vision-Language Models

Abstract

Cross-layer key–value cache sharing diminishes the memory footprint of vision-language models; however, similarity-based acceptance criteria might fail to detect impairments in specific visual capabilities. We conduct an audit to determine whether the statistics employed in the approval process for sharing accurately predict the preservation of object–attribute binding. Utilizing reconstructed and adapted sharing mechanisms, we differentiate between static inter-layer similarity and representation similarity measured on the cumulative modified model. Additionally, we compare acceptance rules under controlled sharing operations and matched cache-reduction budgets. The evaluated gates can admit configurations with significant binding degradation, whereas aggregate visual question-answering and object-presence scores offer an incomplete assessment of the loss. Moreover, evaluating substitutions individually does not establish the retained capability when they are combined. We introduce RecallGate, a lightweight acceptance rule that temporarily incorporates each proposed substitution and evaluates the cumulative configuration on a small visual-recall battery. Matched-budget comparisons demonstrate that capability-based acceptance can preserve substantially more binding capability than proxy-based alternatives. Across various model families and selection strategies evaluated, its effectiveness is contingent upon probe sensitivity and baseline competence. These findings advocate for capability-specific evaluation of compression gates and independent validation of the final shared configuration, while positioning RecallGate as empirical screening rather than a general safety guarantee.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Vision-language models, KV-cache compression, cross-layer KV sharing, object–attribute binding, capability retention, similarity-based acceptance, capability-aware evaluation, RecallGate.

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_xXuqOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether similarity-based acceptance criteria for cross-layer KV-cache sharing can reliably preserve visual capabilities in VLMs. The authors demonstrate that conventional similarity metrics and aggregate VQA scores may fail to capture degradation in object–attribute binding. To address this, they introduce RecallGate, which evaluates each proposed sharing operation using a small capability-specific probe set under the cumulative compressed configuration. Experiments across multiple VLM families show improved binding retention in several matched-compression comparisons, while revealing sensitivity to probe selection and interactions between shared layers.

Critical strengths

The paper highlights an important limitation of conventional compression evaluation: representation similarity and aggregate task performance may not reflect the preservation of specific visual capabilities. The investigation of object–attribute binding provides a useful perspective on compression-induced degradation.

Critical weaknesses

1.The evaluated Q Cache, MiniCache-inspired, and KVSharer-inspired implementations are reconstructed or adapted rather than faithful reproductions. Notably, the Q Cache reconstruction fails to recover the original operating point and exhibits severe degradation in aggregate task performance. Consequently, the conclusions may reflect properties of the adapted implementations rather than limitations of the original methods.

2.RecallGate uses only 16 synthetic binding examples by default, which may provide an unstable acceptance signal. Repeatedly evaluating candidates on the same battery also introduces potential adaptive overfitting. The observed sensitivity to battery size, acceptance thresholds, and random seeds raises concerns about the reliability of the proposed acceptance decisions.

3.The controlled SmolVLM2 experiments reveal substantial performance variation between configurations selected under identical compression budgets. In particular, RecallGate configurations A and B achieve 89.7% and 58.3% layout binding accuracy, respectively. Despite evaluating cumulative configurations, RecallGate can still accept combinations that exhibit severe joint degradation. This limits the robustness of the proposed selection mechanism.

4.Several comparisons rely on limited evaluation samples or fixed selected configurations. Although the authors report confidence intervals for certain comparisons, the controlled selection experiment contains only five runs producing two distinct configurations. Stronger selection-level statistical analyses across additional models and independent probe distributions would improve the reliability of the conclusions.

Questions for authors

1.What is the fundamental methodological contribution of RecallGate beyond replacing conventional similarity-based acceptance with capability-specific validation?

2.How can the authors ensure that the reported findings generalize to the original KV-cache compression methods, given that several evaluated implementations are reconstructed or adapted?

1 comment
AuthorsOpenReview ↗

Response to xXuq: Contribution, Reconstruction Scope and Selection Stability

We thank the reviewer for the positive assessment and for recognizing that representation similarity and aggregate scores can miss capability loss. We ran new experiments on battery stability, real-image transfer, newer models and larger evaluations. Accuracy-difference CIs are paired 95% bootstrap intervals resampling whole images; differences are in percentage points.

Q1. Contribution beyond capability-specific validation. RecallGate is deliberately simple; the contribution is what it reveals and controls. (i) Audit of two tested acceptance criteria: Q Cache's static attention-divergence score has near-zero correlation with retained binding across 104 single shares in four families (family-stratified ρ = +0.009, descriptive), it admitted the most destructive share available in two families, and POPE and GQA stay near baseline while binding collapses; a cumulative hidden-state cosine test, as in KVSharer, admits Qwen2-VL MC-R configurations spanning 73% to 0% binding at one threshold. (ii) Two distinctions the field conflates: single-substitution vs cumulative evaluation, and representation similarity vs capability. The second matters even when the first is handled: cumulative installation with a similarity test still admits severe binding loss. (iii) Controlled evidence that adding a capability check improves selection: at the same threshold, candidate order and number of shared layers, RG keeps more binding at 17/17 matched doses where selections differ (+8 to +69 points; table in our response to juBr). (iv) Selectivity: a spatial-relation battery improves held-out spatial accuracy by +7.0 [+1.3, +13.0] over the proxy but does not protect binding, supporting capability-specific validation, without establishing preservation of either tested or untested capabilities (response to bMQf, Q1).

W1/Q2. Reconstructions and generalization to the original methods. We agree this bounds our claims, and we state it as scope. The audited object is the acceptance decision: our rebuild of Q Cache's criterion reproduces its reported acceptance volumes and is validated component by component, while its published operating point did not reproduce. The 104-share audit evaluates single shares on unmodified models, so it does not depend on that operating point. KS-R and MC-R borrow ranking principles from KVSharer and MiniCache and use a cumulative hidden-state cosine test; we do not evaluate their complete implementations. The results extend across model families and newer checkpoints under our implementation: on Qwen3-VL-2B and InternVL3.5-1B, proxy approval collapses binding at only two shared layers (79.0 → 17.0, 98.3 → 58.3) while RG keeps 77.0 and 90.3. They do not establish that the original methods exhibit the same failures; the revision states this.

W2/W4. Battery size, adaptive overfitting and selection-level statistics. We added selection-level repeats with independent gate seeds:

SettingGate seedsDistinct configurations selected
SmolVLM2 KS-R, original 16-item battery (Table 7)52
SmolVLM2 KS-R, 32-item battery51 (configuration A in 5/5)
SmolVLM2 KS-R, 16 items, half new layouts5A in 4/5
Qwen3-VL-2B / InternVL3.5-1B / 8B3 each1 each
InternVL3.5-2B / 4B3 each2 each
Qwen2-VL, original 16-item battery (Table 7)55

A larger battery stabilizes selection in the tested setting, and the newer models are mostly stable; the Qwen2-VL spread remains and is reported as is. Gate seeds measure selection sensitivity; transfer to evaluations the gate never saw provides generalization evidence. On real-photo GQA colour questions (n=400 on 392 images), RG beats the proxy in all five matched cells, with intervals excluding zero in three (e.g. +7.2 [+3.8, +10.9]); on 491 POPE items, after excluding items whose images were used for calibration, RG wins 4, the interval includes zero in 1, and RG loses 1 (OneVision KS-R). Neither rules out adaptation to the repeatedly reused battery, which remains a risk; RG is a comparative improvement, not a guarantee.

W3. Configurations A and B. Both A and B outperform proxy approval on the original binding evaluation at two shared layers (85.7% and 77.3% versus 23.0%). However, B's 58.3% layout accuracy remains a substantive generalization failure. Enlarging the battery to 32 items selects A in all five tested seeds; this reduces selection of B in that setting, without establishing universal robustness.

We thank the reviewer again, and hope these results strengthen confidence in the paper's conclusions.

Reviewer_juBrOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper audits acceptance criteria for cross-layer KV sharing in VLMs. It asks whether similarity-based proxies (hidden-state cosine, JS divergence) predict retention of object–attribute binding. The authors propose RecallGate: each candidate share is installed on the cumulative configuration and kept only if it scores at least 0.9× the unmodified model on a 16-item binding battery.

Critical strengths
  1. The problem is relevant. The paper shows cases where POPE/GQA stay near baseline while binding drops from 80% to 16%, a useful warning for compression evaluation.
  2. The reporting is honest. Negative results are disclosed openly: the unrecovered Q Cache operating point, the transfer failure of configuration B, and no gain on LLaVA.
Critical weaknesses
  1. Limited novelty. KVSharer already installs candidates cumulatively, evaluates them, and reverts failures. The contribution reduces to swapping the acceptance statistic for a task-specific battery, essentially “validate on the target metric.”
  2. Confounded comparisons. In the matched-threshold comparisons, RecallGate generally shares fewer layers, so much of its gain may simply reflect lighter compression. Qwen2-VL uses different thresholds for the two rules (0.015 vs. 0.03). Only two comparisons, on SmolVLM2 and OneVision, are cleanly controlled.
  3. Overfitting and generalization risk. The gating battery and the evaluation share template families. Some RecallGate scores exceed the healthy model (e.g., 91.3 vs. 80.0), which hints at battery adaptation or noise. Configuration B passes the gate yet drops to 58.3% on layouts, which undermines confidence in the method itself.
  4. Possible harm to other capabilities. The gate protects only binding. On SmolVLM2, RecallGate lowers POPE from 80 to 70–73.3, and cross-capability trade-offs are not analyzed systematically.
Questions for authors
  1. Can you report binding-vs.-savings curves for proxy approval and RecallGate obtained by sweeping thresholds, so that the two rules are compared at equal compression dose?
  2. Does RecallGate’s advantage hold on a real-image binding benchmark, rather than on the synthetic generator shared by the gating battery and evaluation?
  3. Would a held-out split of the gating battery, or a battery with more layout diversity, have rejected configuration B?
  4. How do the conclusions change if the original Q Cache / KVSharer code is used instead of the reconstructions?
2 comments
AuthorsOpenReview ↗

Response to juBr (1/2): Novelty, Dose Control and Generalization

We thank the reviewer for recognizing the problem's relevance and our transparent reporting. Accuracy-difference CIs are paired 95% bootstrap intervals resampling whole images, in percentage points. Part 1 answers the four weaknesses; Part 2 answers Q1 to Q4.

W1. Novelty. We agree that cumulative installation and rollback are existing procedures. Our contribution is the empirical audit of two tested similarity-based acceptance criteria, the distinction between representation change and capability preservation, and controlled evidence that adding a capability check improves selection. Audit: across 104 single shares in four families, the pooled family-stratified Spearman correlation between Q Cache's static attention-divergence score and retained binding is near zero (ρ = +0.009, reported descriptively); in two families this criterion admitted the most destructive share available to it, while POPE and GQA stay near baseline as binding collapses. Cumulative criterion: in our KS-R and MC-R arms, proxy approval already installs each candidate on the cumulative configuration and thresholds final-hidden-state cosine change, as KVSharer does. Along the proxy's Qwen2-VL MC-R trajectory at ε = 0.03, configurations admitted by this cumulative test span 73% binding (1 shared layer) to 0% (4 shared layers; table in W2). Cumulative installation with a similarity test therefore does not establish binding preservation. Scope: KS-R and MC-R borrow ranking principles from KVSharer and MiniCache; they do not evaluate the original methods' complete implementations.

W2. Confounded comparisons. Both rules accept layers greedily, so the first n acceptances of each run are exactly its configuration at n shared layers. Within a model every decoder layer stores the same amount of KV data, so equal counts mean equal theoretical KV-cache storage savings under our implementation; we never compare counts across models. In these trajectories both rules use the same threshold (ε = 0.03 on Qwen2-VL, not the earlier 0.015 vs 0.03 pairing), the same candidate order and the same calibration data; they differ only in RG's added check. Binding %, proxy → RG, mean of 3 evaluation draws of 100 items:

Shared layers123456
Qwen2-VL KS-R40.0→74.333.3→71.739.0→58.7
Qwen2-VL MC-Rsame (72.7)56.0→78.338.3→75.30.0 / RG stopped at 3
OneVision KS-Rsame61.0→69.353.7→67.744.7→64.736.7→53.720.7→41.3
OneVision MC-Rsame63.3→78.761.7→72.70.0→69.0
SmolVLM2 KS-R, A34.7→86.723.0→85.7
SmolVLM2 KS-R, B34.7→78.723.0→77.3

Seven RG trajectories give 22 matched doses; SmolVLM2 A and B are two gate-seed trajectories sharing one proxy trajectory. At 5 doses both rules chose identical layers (the three "same" cells, and both shares of SmolVLM2 MC-R, not shown); at the other 17, RG keeps more binding in 17/17 (+8 to +69 percentage points). Points within a trajectory are related, not independent. The Qwen2-VL MC-R dose-4 entry is proxy-only because RG stopped at three shares; it is excluded from the 22 matched-dose comparisons. These comparisons control dose and threshold, so the advantage is not explained by lighter compression.

W3. Overfitting and generalization. On real-photo GQA colour questions (n=400 on 392 images, never used in selection), RG is higher than the proxy in all five matched cells, with intervals excluding zero in three (Q2); this is the off-template evidence. V-BIND without filler context removes the retrieval component within the synthetic family, and RG beats the proxy there in 5/5 cells. We do not claim RG scores above the unmodified model as gains. Configuration B: a 32-item battery selects configuration A in 5/5 gate seeds and never B, and a 16-item battery with half new layouts selects A in 4/5 (Q3). Increasing battery size or layout diversity reduces selection of B in the tested setting; it does not eliminate the generalization risk.

W4. Harm to other capabilities. On 501 POPE items the unmodified SmolVLM2 scores 87.4, RG configurations A and B 86.4 and 85.0, and proxy approval 78.8. The larger evaluation shows smaller observed baseline gaps than the original 30-item estimate, alongside improvements over proxy approval (+7.9 [+4.6, +11.3] and +6.5 [+1.8, +11.3] on the 491 items whose images were not used for calibration). Across POPE comparisons RG wins 4, the interval includes zero in 1, and RG loses 1 (OneVision KS-R, -20.8 [-26.0, -15.8], where the model answers "yes" to almost every item). A binding battery cannot certify capabilities it does not test; a spatial battery improves held-out spatial accuracy by +7.0 [+1.3, +13.0] over the proxy (see our response to Reviewer bMQf, Q1).

AuthorsOpenReview ↗

Response to juBr (2/2): Threshold Sweeps, Real-Image Binding, Battery Design and Original Code

Q1. Binding-vs-savings curves at equal compression dose. We provide two separate experiments. (a) Threshold sweep, Q Cache reconstruction (paper, main results table and appendix frontier figure): both rules are run at two thresholds per family (Qwen2-VL and OneVision ε = 0.03 / 0.08; SmolVLM2 ε = 0.008 / 0.015). Savings here are the theoretical KV-cache storage reduction n/(2L), assuming equal-sized key and value caches and one key cache removed per share, not measured GPU memory. In three of four families, RG at the higher threshold saves more and retains more binding than the proxy at the lower threshold, while the proxy's higher-threshold savings come with binding collapse:

Family (savings, binding %)Proxy, low εProxy, high εRG, high ε
Qwen2-VL8.9% saved, 49.739.3%, 9.019.6%, 88.3
OneVision2.1%, 37.020.8%, 0.010.4%, 62.7
SmolVLM28.3%, 32.020.8%, 0.010.4%, 76.0

LLaVA-1.5 is the exception: proxy approval and RG score 31.0% and 29.0% binding at ε = 0.008, so this setting shows no observed RG improvement. (b) Equal-dose trajectories, KS-R and MC-R (Part 1, W2): the tested threshold settings do not provide equal-dose comparisons, so we compare the rules at every equal number of shared layers under a common threshold; RG keeps more binding at 17/17 matched doses where the rules differ (+8 to +69 percentage points). These are prefixes of greedy runs, not a dense threshold sweep.

Q2. Real-image binding. On real-photo GQA colour questions (n=400 on 392 images), never used for gate tuning or selection (accuracy %; paired bootstrap 95% CI resampling whole images):

Matched cell (shared layers)Unmod.ProxyRGRG minus proxy [CI]
SmolVLM2 KS-R (2)56.849.054.8+5.8 [+1.7, +10.0]
Qwen2-VL KS-R (3)59.050.051.0+1.0 [-2.5, +4.5]
Qwen2-VL MC-R (3)59.046.854.0+7.2 [+3.8, +10.9]
OneVision KS-R (6)58.226.030.8+4.8 [+0.0, +9.6]
OneVision MC-R (4)58.247.554.2+6.8 [+2.0, +11.6]

RG has higher observed accuracy in all five comparisons, with intervals excluding zero in three. On the relational subset ("the cup left of the plate"; n=200), RG is again higher in all five, with intervals excluding zero for Qwen2-VL MC-R (+7.0 [+2.0, +12.3]) and OneVision KS-R (+9.5 [+2.5, +16.6]). The advantage therefore extends beyond the synthetic generator. These are object-colour questions rather than an isolated binding test, and RG remains below the unmodified model.

Q3. Would a held-out or more diverse battery reject configuration B? In the setting where B was selected (SmolVLM2, KS-R), we re-ran selection over 5 gate seeds. A 32-item battery selects configuration A in 5/5 seeds and never selects B; a 16-item battery with half new layouts selects A in 4/5. Larger or more diverse batteries reduce selection of B in this setting. This shows that B is not selected, not that B would fail these batteries if audited directly. We did not run a held-out split of the original 16-item battery.

Q4. Original Q Cache and KVSharer code. We audit reconstructions, not released code. The Q Cache arm uses its static criterion, the mean last-row attention JS divergence between adjacent layers of the unmodified model (also the score in the 104-share audit); our rebuild reproduces Q Cache's reported acceptance volumes (paper, reproduction ledger) and is validated component by component (score computation, block formation, projection reuse in prefill and decode, calibration responses), although Q Cache's published operating point did not reproduce. No reference implementation was available to us for the visual-only splicing variant, so we did not evaluate that variant. KS-R and MC-R port the ranking principles of KVSharer and MiniCache (developed for language models) and approve candidates with a cumulative final-hidden-state cosine test, analogous to KVSharer's acceptance step; MiniCache's merge is not implemented. We have not run either method's official code. Official implementations could change which layers are selected and the size of the damage; the revision states this limitation.

We thank the reviewer again. The new results control dose and threshold together (17/17 matched doses), show transfer to real images, reduce selection of configuration B with larger batteries, and supplement the original 30-item POPE evaluation with a 501-item evaluation. We respectfully ask the reviewer to reconsider the rating, and welcome further questions.

Reviewer_bMQfOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies post-hoc cross-layer KV-cache sharing in vision-language models (VLMs). Post-hoc sharing methods rely on a similarity score to accept or reject the sharing of the KV cache between layers, and the paper asks whether this score indicates the loss of object-attribute binding, i.e., the ability of the model to connect an object to its attribute. Across experiments on five checkpoints from four VLM families, the authors find only a weak relation between the similarity scores and binding loss. They further show that aggregate benchmarks such as POPE and GQA can retain their performance while binding accuracy collapses.

Building on this observation, the paper introduces RecallGate, which adds a second acceptance step to existing KV-cache sharing algorithms. A sharing step is accepted only if it passes the original similarity threshold and the modified model also retains at least 90% of its original binding accuracy on a small synthetic test set. RecallGate yields shared models with better binding scores than acceptance based on the similarity threshold alone. However, the results vary across random seeds, and the gated models do not consistently keep the binding accuracy of the unmodified model on new image layouts.

Critical strengths

S1. A relevant and under-studied question with useful conceptual distinctions. Rather than reporting only the compression ratio and aggregate accuracy, the paper examines whether the acceptance criteria for KV-cache sharing are valid in the first place. This question has clear practical relevance, since post-hoc KV-cache sharing methods depend entirely on such criteria. The paper also separates two questions that are easy to conflate: whether a criterion evaluates a single substitution or the cumulative configuration, and whether it measures representation similarity or the target capability itself. These distinctions can be reused to analyze other compression criteria.

S2. Evidence that existing criteria accept sharing with hidden changes in behavior. The similarity scores do not predict binding loss, and aggregate benchmarks do not reveal it. Existing criteria can therefore accept sharing configurations that change model behavior in ways that standard evaluation does not detect.

S3. Honest reporting of negative results. The authors report negative results alongside positive ones. For example, they show that a configuration accepted by RecallGate falls below the unmodified model on new layouts, and they state that they could not reproduce the published Q Cache results. Such reporting is unusual in compression papers and is valuable for the field.

Critical weaknesses

W1. The main result is circular. RecallGate and the evaluation both use synthetic binding probes from the same template families. The layout variants in Section 5.5 do not resolve this, because they keep the same three shapes, six colors, and color question, and change only the spatial arrangement. The paper contains no binding evaluation on real images.

W2. The choice of binding is not justified. The paper does not argue why binding matters in practice, nor why it is preferred over other capabilities such as counting, spatial relations, OCR, or localization. No link to a practical use case is shown. Moreover, the V-BIND-NIAH test does not measure binding in isolation, since the model must also locate the correct image within a long text with filler context. Finally, the generality of the method is demonstrated only through the POPE ablation, and even there POPE is not fully preserved (76.7% versus 90.0%).

W3. RecallGate is not reliable. The reported results change with the gating seed (Table 7), and configurations approved by RecallGate can still fail, even though the gate and the evaluation share the same probe family. As the authors themselves report, configuration B in Section 5.5 falls below the unmodified model on new layouts. In addition, the accepted OneVision configuration in Table 4 retains only about 49% of its binding accuracy, despite the 90% acceptance floor.

W4. The model selection is outdated. The experiments use LLaVA-1.5 (2023), Qwen2-VL and LLaVA-OneVision (2024), and SmolVLM2 (2025). Three of the five checkpoints have 2.2B parameters or fewer, and the 13B model appears only as a scale check with a single evaluation draw. Current open VLMs that were available before submission, such as Qwen3-VL, Molmo2, InternVL3.5, or Gemma 4, are missing. Beyond the two LLaVA-1.5 sizes, the paper also lacks a systematic scale study within a single family (e.g., InternVL3.5 from 1B to 14B), as well as a reference model with trained cross-layer sharing (e.g., Gemma 4 E2B/E4B).

W5. Presentation. Some parts of the paper are hard to read and would benefit from a rewrite. Several terms are used without definition.

Questions for authors

Q1. Why did you select object-attribute binding as the target capability rather than another capability such as counting, spatial relations, OCR, or localization? Is there a practical application in which the loss of binding leads to a failure? Could you show results for a second target capability?

Q2. Since the gate and the evaluation use the same type of synthetic binding probe, how do you show that RecallGate selects better layers in general and does not merely fit the probe family?

Q3. Do the results hold for current VLMs such as Qwen3-VL, Molmo2, InternVL3.5, or Gemma 4? How does model size affect RecallGate, for example across the InternVL3.5 sizes from 1B to 14B? How does binding retention under post-hoc sharing compare with models that use trained cross-layer sharing, such as Gemma 4 E2B/E4B?

2 comments
AuthorsOpenReview ↗

Response to bMQf (1/2): Real-Image Transfer and Binding Retention

We thank the reviewer for recognizing the criterion-validity question (S1), the hidden-degradation evidence (S2) and our negative results (S3). The central contribution is an empirical audit of whether similarity-based acceptance tracks capability preservation under post-hoc KV-cache sharing; RecallGate (RG) tests whether adding a direct capability check improves layer selection. All proxy-versus-RG comparisons use the same number of shared layers and identical items. All reported confidence intervals are paired 95% bootstrap intervals, resampling whole images. Part 1 answers W1 to W5; Part 2 answers Q1 to Q3.

W1. Circularity. We evaluated the selected configurations on real-photo GQA colour questions ("what colour is the car?"; n=400 on 392 images), which share no template with the gate and were never used for gate tuning or selection (accuracy %):

Matched cell (shared layers)Unmod.ProxyRGRG minus proxy [CI]
SmolVLM2 KS-R (2)56.849.054.8+5.8 [+1.7, +10.0]
Qwen2-VL KS-R (3)59.050.051.0+1.0 [-2.5, +4.5]
Qwen2-VL MC-R (3)59.046.854.0+7.2 [+3.8, +10.9]
OneVision KS-R (6)58.226.030.8+4.8 [+0.0, +9.6]
OneVision MC-R (4)58.247.554.2+6.8 [+2.0, +11.6]

RG has higher observed accuracy in all five comparisons, with image-level paired bootstrap intervals excluding zero in three. These results provide evidence that the benefit extends beyond the gate's synthetic template family. On the relational subset ("the cup left of the plate"; n=200), supporting analysis only, RG is higher in all five, with CIs excluding zero for Qwen2-VL MC-R (+7.0 [+2.0, +12.3]) and OneVision KS-R (+9.5 [+2.5, +16.6]). This is object-colour QA, not an isolated binding test. (These values come from the run that saved per-item answers; an earlier run of the same configurations differed by at most 0.4 percentage points.) Complementary: on a 501-item POPE subset, after removing all 10 questions on the 8 images used to calibrate the similarity score (n=491 on 311 images), RG beats the proxy on SmolVLM2 KS-R (+7.9 [+4.6, +11.3]; config B +6.5 [+1.8, +11.3]), Qwen2-VL KS-R (+10.0 [+6.1, +14.0]) and OneVision MC-R (+14.7 [+9.2, +20.0]), shows no significant difference on Qwen2-VL MC-R (-2.2 [-5.1, +0.6]), and loses on OneVision KS-R (-20.8 [-26.0, -15.8]), where the model answers "yes" to almost every item: a binding-only gate can miss other degradation.

W2. Binding, NIAH, POPE. Why binding: Q1. NIAH: V-BIND without filler context (n=100) shows the same pattern, RG above the proxy in 5/5 matched cells (Q2), so retrieval difficulty alone cannot explain the degradation. POPE 76.7 vs 90.0: gating on POPE admitted 14 Qwen2-VL shares, yet POPE fell; passing a finite battery does not guarantee preservation on held-out items.

W3. Reliability. We distinguish two questions: whether RG selects configurations with better held-out binding than proxy-only approval, and whether it guarantees preservation. The matched-dose experiments support the first: RG has higher binding at all 17 tested doses where the selected layers differ (+8 to +69 percentage points), across 6 sharing sequences (5 further doses had identical layers). We compare only doses reached by both rules; points within a sequence are related. The 90% threshold is an acceptance criterion on the gate battery, not a guarantee on unseen data; the OneVision 49% retention does not contradict the rule but shows a substantial generalization gap, stated in the revision. Binding %, mean ± SD over 3 evaluation draws of 100 items:

Shared layers1234
Qwen2-VL KS-R, proxy40.0±2.033.3±1.539.0±3.628.0±6.1
Qwen2-VL KS-R, RG74.3±2.571.7±4.558.7±2.5stopped
OneVision MC-R, proxysame63.3±4.261.7±3.10.0±0.0
OneVision MC-R, RGsame78.7±2.572.7±5.869.0±2.6

Gate seeds (separate from the ± SD): in the Sec. 5.5 setting, a 32-item battery selects config A in 5/5 seeds and never B; this does not establish stability for all models, and Table 7's Qwen2-VL spread is reported as is. The main finding is that passing the tested similarity criteria does not reliably establish preservation of object-attribute binding; RG is a comparative improvement, not a guarantee.

W4. Models. We added Qwen3-VL-2B and a within-family scale study of InternVL3.5 at 1B, 2B, 4B and 8B (Q3).

W5. Presentation. The revision defines terms at first use and adds a glossary: proxy approval, accept a share if its similarity score passes ε; RG, proxy approval plus keeping at least 90% of unmodified binding on a small battery (16 items by default); KS-R / MC-R, dissimilarity- / similarity-ranked candidate order (KVSharer- / MiniCache-style); dose, number of shared layers.

AuthorsOpenReview ↗

Response to bMQf (2/2): Second Capability, Generalization and Current Models

Q1. Why binding, a practical failure, and a second capability. We chose binding as a focused test of whether compression preserves associations between recognized objects and attributes, rather than only their individual recognition. Grounded requests depend on it ("which car is red?", "which of two medication bottles carries the warning label?"), and its loss is what aggregate benchmarks hide (S2): SmolVLM2 under KS-R retains 100% of baseline POPE accuracy but 16% of baseline binding accuracy. For a second capability we instantiated the same acceptance mechanism with a spatial-relation battery: GQA "left or right of" questions on real images, gate and test images disjoint, answer order balanced (Qwen2-VL-2B, 4 shared layers; n=300 on 284 images; paired 95% bootstrap CIs resampling whole images; differences in percentage points, computed before rounding):

ConfigurationSpatial acc. %vs proxyvs unmodified
Unmodified73.3
Proxy approval60.7-12.7 [-20.3, -4.9]
Spatial RG67.7+7.0 [+1.3, +13.0]-5.7 [-12.5, +1.0]

Spatial RG recovers about 55% of the observed proxy-induced loss. Its accuracy remains 5.7 percentage points below the unmodified model; the CI includes zero and does not establish equivalence. These results support capability-specific checks: a spatial battery improves held-out spatial accuracy, while neither battery establishes preservation of capabilities it does not evaluate (the spatial gate leaves V-BIND at 34.7; the binding gate does not protect POPE on OneVision KS-R, W1). Boundary: one model and one gate seed; counting (our probe was unreliable), OCR and localization remain untested.

Q2. Better layers in general, or a fit to the probe? We distinguish generalization of the binding-selected configurations from generality of the gating mechanism. The former is evaluated through held-out GQA colour questions (higher in 5/5, CIs excluding zero in 3; W1), POPE with calibration images removed (4 wins, 1 interval includes zero, 1 loss; W1), and V-BIND without filler context (binding %, n=100, single runs):

Matched cell (shared layers)Unmod.ProxyRG
SmolVLM2 KS-R (2)962891
Qwen2-VL KS-R (3)661657
Qwen2-VL MC-R (3)663245
OneVision KS-R (6)761738
OneVision MC-R (4)761763

Separately, Q1 shows that the mechanism also improves held-out accuracy with a spatial battery. Boundary: the results support transfer beyond the synthetic templates in several settings, without establishing universal preservation or excluding some adaptation to the gate battery.

Q3. Current VLMs, model size, and trained sharing. Binding % at 2 shared layers; each entry, including each RG gate seed, is mean ± SD over the same 3 evaluation draws:

ModelUnmod.ProxyRG, gate seeds 1 / 2 / 3
Qwen3-VL-2B79.0±2.017.0±3.677.0±2.0 / 76.7±2.5 / 77.0±2.0
InternVL3.5-1B98.3±0.658.3±2.590.3±0.6 / 90.0±1.0 / 90.0±0.0
InternVL3.5-2B99.7±0.696.3±3.196.3±3.1 / 96.3±3.1 / 98.3±1.1
InternVL3.5-4B99.7±0.693.7±2.9100±0.0 / 93.7±3.1 / 100±0.0
InternVL3.5-8B100±0.099.3±0.699.3±0.6 / 99.3±0.6 / 99.3±0.6

Current models: similarity-based approval accepts configurations that collapse binding (Qwen3-VL-2B: 79.0 to 17.0; InternVL3.5-1B: 98.3 to 58.3), and RG recovers most of it. Selection stability: on these two models all three gate seeds selected identical source-to-target layer mappings. Execution reproducibility (separate): decoding is greedy and computation is in bfloat16. For Qwen3-VL-2B, the two runs on the same GPU type gave identical answers on all 300 items, while a run on a different GPU type changed the correctness of 11; for InternVL3.5-1B (up to 7 changed) the GPU type was not logged. Mean accuracy varied by at most 0.3 percentage points. Size: the large observed accuracy improvements on Qwen3-VL-2B and InternVL3.5-1B are substantially larger than the aggregate accuracy variation observed in these repeated evaluations. For InternVL3.5-2B to 8B we report results descriptively and claim no reliable RG advantage. Shared layers are held fixed (2 of 28 language-decoder layers at 1B and 2B, 2 of 36 at 4B and 8B), so size is not isolated from architecture; higher doses remain untested. Trained sharing: a trained-sharing model would provide a useful reference for binding retention, although it would not directly test a post-hoc acceptance criterion. We have not performed that comparison and make no claim about relative retention. Boundary: InternVL3.5-14B, Molmo2 and higher doses were not run; the revision states this limitation.

We hope the additional evidence and clarified scope address the reviewer's concerns, and respectfully ask the reviewer to reconsider the rating.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response: Summary of New Evidence and Scope of Conclusions

Dear Reviewers and Area Chair,

Thank you for your thoughtful reviews and constructive feedback. We appreciate the shared recognition that representation similarity and aggregate benchmarks can miss compression-induced capability loss. We have provided detailed individual responses and additional experiments. Below, we summarize the main evidence and clarify the scope of our conclusions.

Central contribution and implementation scope. Our main contribution is an empirical audit of two tested similarity-based acceptance criteria, together with controlled evidence that adding a direct capability check improves layer selection. RecallGate is deliberately simple; cumulative installation and rollback are not claimed as novel. Our experiments use reconstructed or adapted implementations: the Q Cache arm uses its static attention JS-divergence criterion, and the KS-R and MC-R arms use a cumulative final-hidden-state cosine criterion, as in KVSharer, with ranking principles borrowed from KVSharer and MiniCache rather than their complete original implementations. Accordingly, our findings concern the tested criteria and implementations; they do not establish that the original methods exhibit identical failures.

Reviewer bMQf: real-image transfer, capability choice, and current models. To address template overlap, we evaluated configurations on real-photo GQA colour questions excluded from gate tuning and selection: 400 questions on 392 images. RecallGate has higher observed accuracy than proxy approval in all five matched comparisons, with image-level paired bootstrap intervals excluding zero in three. These questions evaluate object-colour associations in real images, rather than binding in isolation. We also instantiated the mechanism with a spatial-relation battery, using disjoint gate and test images. On 300 held-out questions, spatial RecallGate improves accuracy over proxy approval by 7.0 percentage points [1.3, 13.0]. New experiments on Qwen3-VL-2B and InternVL3.5 checkpoints from 1B to 8B broaden model coverage. At two shared layers, proxy approval reduces Qwen3-VL-2B binding from 79.0% to 17.0%, whereas RecallGate retains approximately 77%. Larger-model results are reported descriptively, without claiming an isolated effect of model size.

Reviewer juBr: novelty, compression controls, and cross-capability effects. We distinguish two-threshold experiments under the Q Cache reconstruction from equal-dose trajectories under KS-R and MC-R. In the latter, both rules use the same similarity threshold, candidate order, and calibration data. RecallGate retains more binding at all 17 matched doses where selections differ, by 8 to 69 percentage points. The trajectories contain related observations, including two gate-seed trajectories sharing one proxy trajectory; these are not 17 independent experiments. Equal shared-layer counts represent equal theoretical KV-cache storage savings under our implementation, rather than measured GPU-memory savings. A larger POPE evaluation supplements the original 30-item estimate. After excluding calibration-image questions, the 491-item comparisons yield four intervals favouring RecallGate, one including zero, and one favouring the proxy. The substantial OneVision KS-R failure is retained and discussed.

Reviewer xXuq: battery stability, adaptive overfitting, and configuration robustness. Enlarging the SmolVLM2 battery from 16 to 32 items selects configuration A in all five tested gate seeds; a 16-item battery incorporating new layouts selects A in four of five. These results reduce selection of configuration B in the tested setting, but do not demonstrate that B would fail a direct audit or eliminate adaptive overfitting. We separate selection stability, measured across gate seeds, from execution reproducibility. We disclose repeated-evaluation differences, including observed cross-GPU variation, and retain the unresolved Qwen2-VL selection spread. Configuration B's layout failure remains evidence of a substantive generalization gap.

The additional results support RecallGate as a comparative improvement in the tested settings, rather than a guarantee of capability preservation. We retain negative results and explicitly identify untested comparisons, including trained-sharing reference models and original implementations. The revised manuscript consolidates these experiments, clarifies definitions and implementation scope, and documents the evaluation and reproducibility procedures; text added during the discussion period is shown in blue.

We appreciate the reviewers' and Area Chair's consideration and hope that the additional evidence and clarified claims support a reassessment of the submission. We welcome further questions.

Best regards, Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.