Auditing Capability Retention under Cross-Layer KV Sharing in Vision-Language Models

Cross-layer key–value cache sharing diminishes the memory footprint of vision-language models; however, similarity-based acceptance criteria might fail to detect impairments in specific visual capabilities. We conduct an audit to determine whether the statistics employed in the approval process for sharing accurately predict the preservation of object–attribute binding. Utilizing reconstructed and adapted sharing mechanisms, we differentiate between static inter-layer similarity and representation similarity measured on the cumulative modified model. Additionally, we compare acceptance rules under controlled sharing operations and matched cache-reduction budgets. The evaluated gates can admit configurations with significant binding degradation, whereas aggregate visual question-answering and object-presence scores offer an incomplete assessment of the loss. Moreover, evaluating substitutions individually does not establish the retained capability when they are combined. We introduce RecallGate, a lightweight acceptance rule that temporarily incorporates each proposed substitution and evaluates the cumulative configuration on a small visual-recall battery. Matched-budget comparisons demonstrate that capability-based acceptance can preserve substantially more binding capability than proxy-based alternatives. Across various model families and selection strategies evaluated, its effectiveness is contingent upon probe sensitivity and baseline competence. These findings advocate for capability-specific evaluation of compression gates and independent validation of the final shared configuration, while positioning RecallGate as empirical screening rather than a general safety guarantee.