acceptodds
Under review as a conference paper at ICLR 2027

KVCapsule: Efficient Sequential KV Compression for VLMs with Asymmetric Redundancy

Abstract

Visual tokens can dominate the key-value (KV) cache of long-context vision-language models (VLMs), increasing persistent memory and decoding traffic. Many visual sequence-reduction methods prune or merge positions using importance estimates obtained before or early in decoding, making discarded KV states unavailable if their relevance changes later. We show that visual KV exhibits sequence-level redundancy, decoding-dependent attention, and distinct redundancy structures for keys and values. Based on these observations, we introduce KVCapsule, an asymmetric visual-KV compression framework that reduces persistent cache storage while maintaining an approximate full visual-position representation for subsequent attention. KVCapsule reconstructs visual keys from selectively retained positions and compresses visual values using sequence-level PCA, while keeping the pretrained VLM backbone frozen. Across VLM backbones and image and video benchmarks, KVCapsule achieves a strong quality-memory trade-off. At 10K context and batch size 1, it reduces the measured persistent KV-related footprint by 55.0% and improves decoding throughput by 24.0%; at larger-batch long-context settings, throughput improves by up to , while our analytical memory model predicts up to a smaller persistent footprint beyond the break-even regime.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Vision-Language Models, KV Cache Compression, Efficient Inference, Multimodal Inference, Long-Context Inference

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Xg2cOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper first investigates the redundancy of vision tokens in VLMs. It then proposes KVCapsule, which compresses keys and values asymmetrically. Keys are selected with a learned, fixed sequence mask and reconstructed to the original sequence length using a two-layer MLP. Values are centered and projected through a learned sequence-dimensional PCA basis, then reconstructed by linear re-projection.

Critical strengths
  1. Substantial investigative experiments. The paper includes a substantial amount of experiments to first investigate and quantify the redundancy of vision tokens in VLMs.
  2. Following the above, I believe the method is well motivated and grounded on prior investigative experiments. Specifically, the asymmetrical compression of keys and values is grounded and novel.
Critical weaknesses
  1. While the investigative experiments provide useful insights in understanding the redundancy of vision tokens, the underlying mechanism behind the observed key value asymmetry remains insufficiently explored. Since keys and values are derived from the same underlying token representations, why do keys exhibit substantially higher inter-token similarity (particularly in earlier layers), while values remain more token-specific (Fig. 4)? A deeper explanation of this phenomenon would strengthen the motivation behind the asymmetric compression.
  2. Missing details in method. The paper provides calibration procedures in Appendix C, but the provenance of the calibration dataset and its relationship to the evaluation datasets remain insufficiently specified. In particular, are the 500 calibration samples strictly disjoint from all downstream evaluation samples? Moreover, how sensitive are the learned operators to the calibration distribution? Since the method relies on fixed, learned reconstruction operators, the authors should investigate and provide evidence of generalization across unseen data distributions.
  3. The efficiency evaluation remains insufficiently characterized. In particular, Fig. 5(c) shows that attention latency under Full KV increases with input length, whereas KVCapsule exhibits a slightly decreasing trend despite maintaining a fixed compression configuration. This behavior is counterintuitive and warrants further explanation. The authors should clarify the workload composition, how they were measured, and sources of latency variation. Preferably, the authors should also report variability across repeated measurements.
  4. The method introduces additional score-space computations, including an O(n2)O(n^2) term per visual block (Sec 4.3: since d_b = n/4, O(rldhead+rldb+ndb)O(r_l d_\text{head} + r_l d_b + n d_b) is essential O(n2)O(n^2) ) . The authors should provide a more detailed analysis of this overhead and clarify how reconstruction costs scale with visual-token length and different tokenization configurations. In particular, it would be useful to identify in which configs the reconstruction overhead outweighs the benefits of reduced KV-memory traffic.
  5. Need for in-depth analysis of reconstruction. App. G reports attention pattern cosine similarities of 0.58, 0.76, and 0.60 for K-only, V-only, and joint K+V compression. However, I think the experimental protocol and interpretation of these values are insufficiently explained. In particular, a similarity of 0.60 suggests substantial deviations from full-KV attention patterns, despite the relatively small downstream performance differences reported in Tables 2 and 3. Furthermore, App R reports a PPL increase of 2.23x relative to full KV, raising additional questions about output quality. The authors should investigate these discrepancies and provide a more comprehensive analysis of how reconstruction errors propagate to attention outputs and final generation quality.
  6. While Tables 2 and 3 provide extensive downstream benchmark comparisons, corresponding efficiency measurements are largely deferred to the appendices. Although Appendix K and P provide useful comparisons, the multi-budget evaluation primarily considers PruMerge. Given the efficiency-focused nature of the paper, a more comprehensive comparison against alternative compression methods under matched memory budgets, reporting both downstream performance and inference efficiency, would strengthen the experimental evaluation.

Overall, the paper produces strong investigative evidence and a well motivated approach. However, several important aspects of the methodology and experimental evaluation remain insufficiently explained.

Questions for authors

See weaknesses.

3 comments
AuthorsOpenReview ↗

Response to Reviewer Xg2c

We thank the reviewer for the questions and feedback.

W1. Why keys and values exhibit different inter-token structure

Under the visual-only MEGABench protocol in Appendix A, visual-key inter-token cosine similarity is 0.99/0.87/0.660.99/0.87/0.66 at layers 0/10/200/10/20, compared with 0.35/0.40/0.300.35/0.40/0.30 for visual values. Across the same layers, values require 113--114 of 128 dimensions to preserve 95% variance and 125 dimensions to preserve 99%, showing that their information remains dense across feature channels.

At the same 60.2% visual-KV reduction, learned key reconstruction plus value PCA gives the best compressed result on five of six metrics; symmetric PCA is slightly higher only on MMMU (0.4860.486 versus 0.4800.480). Together, the representation measurements and matched-memory ablation show that the structural difference translates into stronger downstream performance at a fixed compression level.

W2. Calibration data, reuse, and source sensitivity

Appendix C specifies that the calibration set contains 500 single-image examples, that operators are fitted independently for each layer and KV head, and that the resulting bank is frozen and reused across image and video evaluations. The submitted bank is fitted on uncompressed visual K/V activations from 500 MEGABench images; no downstream labels, answers, scores, or evaluation examples are used. Appendix Q further evaluates reuse of a fixed bank across inputs containing 8 to 64 video frames. We will consolidate these details next to the main experimental setup.

To directly test sensitivity to the calibration source, we separately fitted c60 banks using 400 MEGABench images and 400 MLVU videos while keeping the model, fitting procedure, downstream manifests, decoding, and evaluator fixed:

Calibration bankCalibration sourceLLaVABench Relative Score ↑\uparrow (60)MMBench-Video Overall ↑\uparrow (639)
ImageMEGABench84.91.09
VideoMLVU85.61.15

The image- and video-derived banks differ by only 0.7 on LLaVABench and 0.06 on MMBench-Video. Thus, in both evaluations, KVCapsule transfers between image- and video-derived calibration banks without benchmark-specific or per-input fitting.

W3. Runtime crossover under repeated measurement

The submission reports an attention-path profile at 9K--15K and a 24.0% end-to-end throughput gain at 10K. We further measure whole-model decoding across context lengths and batch sizes using 117-token visual blocks and the same c60 configuration. Each cell uses 20 warmup iterations, 100 synchronized timed measurements, and 128 greedy output tokens on one AMD MI250X. We report mean time per output token (TPOT) with bootstrap 95% confidence intervals over the timed measurements.

BatchContextFull KV TPOT (ms)KVCapsule TPOT (ms)Throughput changePersistent-memory reduction
15K46.264 [46.231, 46.296]48.196 [48.087, 48.306]-4.0%56.30%
110K60.593 [60.585, 60.602]50.166 [49.973, 50.367]+20.8%58.50%
115K75.387 [75.384, 75.391]48.422 [48.371, 48.479]+55.7%59.24%
45K59.628 [59.622, 59.635]48.652 [48.535, 48.776]+22.6%56.81%
410K86.777 [86.769, 86.787]55.997 [55.988, 56.009]+55.0%58.77%
415K114.473 [114.462, 114.485]64.429 [64.427, 64.431]+77.7%59.42%

For 117-token visual blocks, KVCapsule crosses Full KV at 10K for batch 1 and 5K for batch 4, with non-overlapping confidence intervals at all five beneficial points. At batch 1 and 5K, reduced KV traffic does not yet offset the additional reconstruction overhead, producing a 4.0% throughput reduction while still reducing persistent memory by 56.3%.

The small batch-1 TPOT change from 50.17 ms at 10K to 48.42 ms at 15K is reproducible under this protocol. Standard-operator GPU latency can vary with tensor shape, so this local fluctuation does not imply decreasing algorithmic cost. The relevant result is the measured crossover and the substantially flatter long-context scaling: Full-KV TPOT continues to increase with context, whereas KVCapsule's reduced KV traffic increasingly offsets its additional reconstruction overhead.

Memory reduction varies across workloads because text KV remains uncompressed and fixed method state is amortized over different context and batch sizes. Each percentage is therefore measured against its corresponding Full-KV reference.

AuthorsOpenReview ↗

Continued Response to Reviewer Xg2c

W4. Reconstruction overhead and scaling

Section 4.3 derives the fused-path complexity, including the O(n2)O(n^2) score-reconstruction term when db=n/4d_b=n/4, and reports a persistent-memory break-even near L=626L=626. Appendix Q specifies that calibrated operators are shared across repeated blocks with the same visual tokenization.

For batch size BB, number of visual blocks NN, uncompressed and retained lengths nn and rr, head width dd, and element size ss, the persistent visual-cache footprint scales approximately as

SFull≃2BNnds,SKVC≃2BNrds+Sop,S_{\rm Full}\simeq 2BNnds,\qquad S_{\rm KVC}\simeq 2BNrds+S_{\rm op},

where SopS_{\rm op} is shared rather than replicated across blocks. This gives the memory break-even: shared operator state matters most at small BNBN, while the n ⁣→ ⁣rn\!\rightarrow\!r reduction dominates as context or batch grows. Runtime also includes the reconstruction arithmetic identified above; W3 measures when the reduced traffic outweighs that execution cost.

An idealized count based on BF16 cache reads and GQA reuse gives approximately 4.0 FLOP/byte for Full KV and 7.8 FLOP/byte for KVCapsule, both below the MI250X BF16/HBM ridge point of approximately 120 FLOP/byte. Both paths are therefore memory-bound in this regime; KVCapsule reduces visual-cache traffic and raises the attainable memory-roof bound. “Fused” refers to algebraically avoiding materialization of reconstructed full-length K/V. The reported implementation uses standard operations without a custom KVCapsule kernel.

We repeat the W3 protocol using independently calibrated c60 operators for 468-token visual blocks:

Visual tokens/blockBatchProcessed contextFull KV TPOT (ms)KVCapsule TPOT (ms)Throughput changePersistent-memory reduction
46817,78254.481 [54.479, 54.483]49.842 [49.760, 49.927]+9.3%46.23%
468110,13261.367 [61.363, 61.373]49.525 [49.378, 49.694]+23.9%49.70%
468115,30276.542 [76.537, 76.552]49.822 [49.696, 49.957]+53.7%53.57%
46847,78274.399 [74.394, 74.404]52.661 [52.657, 52.665]+41.3%51.36%
468410,13287.300 [87.293, 87.308]56.562 [56.553, 56.577]+54.3%53.64%
468415,302116.183 [116.173, 116.195]65.209 [65.203, 65.219]+78.2%56.19%

KVCapsule is faster in all six settings with non-overlapping TPOT confidence intervals, and its advantage increases with context and batch size. Together with W3, this demonstrates the same scaling behavior for independently calibrated 117- and 468-token visual blocks. The larger method-state term for 468-token operators is included in every reported memory measurement.

W5. Reconstruction fidelity, attention similarity, and perplexity

Appendix R reports output-level fidelity on a fixed deterministic 60-example LLaVABench subset using attention-output SNR, decoding-step logit KL, top-1 agreement, and sequence-level perplexity.

The reported 2.2253×2.2253\times relative PPL is computed by scoring the same 20-token Full-KV continuation under Full KV and KVCapsule. Across 60 examples and 1,200 tokens, the absolute PPL values are 2.04432.0443 and 4.54924.5492, respectively. It therefore measures fidelity to Full KV's exact continuation, whereas Tables 2 and 3 evaluate downstream answer quality.

On this fidelity diagnostic, KVCapsule achieves 18--20 dB attention-output SNR; logit KL changes from 2.0562.056 to 0.7990.799 and top-1 agreement from 0.450.45 to 0.850.85 over decoding steps 1--20. K-only and joint compression also have nearly identical PPL (4.55334.5533 versus 4.54924.5492), while joint compression increases KV reduction from 30.1% to 60.2%. Thus, value compression provides the additional memory reduction without further degradation on this diagnostic.

W6. Matched-memory comparisons

Appendices K and P provide complementary comparisons using measured persistent memory. Appendix K reports memory and throughput for Full KV, FastV, PruMerge, KVCapsule, and SparseVLMs under the same 10K setting. Appendix P compares KVCapsule and PruMerge at approximately 40%, 60%, and 70% measured reduction, including method-specific state and downstream quality. The evaluated SparseVLMs implementation sparsifies attention without compacting stored K/V, so it remains in the 10K efficiency comparison but not the multi-budget cache-compression curve.

The controlled LOOK-M/VisCache evaluation above further compares KVCapsule, LOOK-M, and VisCache at nearby measured reductions of 55.54%, 54.32%, and 57.57%, respectively, across four image and video benchmarks. Together, these experiments provide efficiency, multi-budget, and downstream-quality comparisons using measured memory rather than nominal token-retention ratios.

Thank you again for your valuable feedback. As the discussion period ends on November 18, we would be grateful if you could let us know whether our rebuttal has addressed your concerns.

Reviewer_KTkGOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes KVCapsule, a training-free KV cache compression method for VLMs. Based on the observation that visual keys and values have different redundancy, KVCapsule keeps a subset of visual keys and reconstructs the rest with a linear operator, and compresses visual values with sequence-level PCA. Experiments on image and video benchmarks show that KVCapsule reduces KV memory and improves decoding throughput across different backbones.

Critical strengths
  • The asymmetric design is simple, and since it is fully linear, it can be fused into attention, which gives real memory and throughput gains on hardware.
  • The ablations on asymmetric K/V compression at matched memory and on full-position reconstruction support the design choices.
Critical weaknesses
  • The insights are mostly a combination of known observations. Visual-token redundancy, shifting token importance during decoding, K/V asymmetry, and layer-wise compression budgets have each been shown in prior work (e.g., FastV, Quest, KIVI, KVMerger, PyramidKV). The fixed linear projection along the sequence dimension is also close to Linformer.
  • Some experimental results are questionable. In Table 2, Qwen2.5-VL Full KV scores only 0.20 on MMMU (around random), the same numbers appear across different models and methods, and the compressed model beats Full KV by over 10 points on LLaVA-Llama. In addition, the paper is motivated by shifting token importance but does not compare with dynamic KV selection methods designed for this (e.g., Quest [1], ShadowKV [2]).

[1] Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference

[2] ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference

Questions for authors
  • Could you explain the anomalies in Table 2 and the inconsistent Full KV scores across Tables 2, 15, and 18?
  • Are the cached keys pre-RoPE or post-RoPE? If post-RoPE, how can a fixed linear operator be reused across images at different positions?
  • How does KVCapsule compare with Quest/ShadowKV and VL-Cache/LOOK-M at matched memory?

I'd like to raise my score if the experimental issues are clarified and the missing baselines are added.

4 comments
AuthorsOpenReview ↗

Response to Reviewer KTkG

We thank the reviewer for the questions and feedback.

W1. Relationship to prior observations and Linformer

KVCapsule's novelty lies in a complete representation-and-execution design for the persistent visual KV cache. Unlike KIVI, which addresses channel-wise quantization, KVCapsule compresses along the visual sequence dimension. Unlike visual-token pruning, it reconstructs an approximate full-position domain instead of permanently restricting later queries to retained positions. Unlike standard Linformer, which learns sequence projections jointly with the backbone and performs attention in a reduced sequence domain, KVCapsule calibrates distinct key-reconstruction and value-PCA operators per layer and KV head for a frozen pretrained VLM. Its fused formulation then evaluates the same reconstructed-attention computation without materializing full-length K/V. The contribution is therefore the combination of asymmetric visual-cache compression, recoverable full-position approximation, frozen-backbone calibration, and fused decode execution--not sequence projection alone.

This distinction is supported by two controlled ablations reported in the submission. Under the same persistent-cache setting, approximate full-position reconstruction increases recovered attention mass on late-important positions from 0.320.32 to 0.630.63 and attention-output cosine from 0.360.36--0.510.51 to 0.800.80--0.820.82 relative to retention-only attention. At the same 60.2% visual-KV reduction, the asymmetric learned-key-reconstruction and value-PCA design also achieves the strongest compressed result on five of six metrics. These results isolate the contributions of recoverability and asymmetric K/V treatment within the complete design.

W2. Evaluation scopes and Full-KV values across tables

The submitted captions identify distinct evaluation scopes. Table 2 reports the main VLMEvalKit evaluation, while Tables 15 and 18 each use a 500-example subset shared across the methods within that comparison. Their Full-KV rows are paired references for different evaluation sets, so cross-table equality is not expected. We will repeat the scope in each table title and state explicitly that the two appendix subsets are independently sampled.

The Qwen2.5-VL entries use the same fractional scale as the other rows. The underlying Full-KV MMMU dev aggregate is 0.20670.2067, while FastV's MMBench dev accuracy is 0.00860.0086; their two-decimal presentation as 0.200.20 and 0.000.00 obscures this distinction. We will report these entries with four decimals and include the split and sample count.

Full KV is the paired computational reference, not a guaranteed upper bound on a finite exact-match or judge-based benchmark. Individual output changes can move a task score in either direction; we do not interpret isolated higher compressed scores as a general improvement to the underlying model.

The controlled comparison in W4 additionally removes cross-protocol ambiguity by using one model, common manifests, identical decoding, dataset-specific evaluation, and one shared Full-KV reference.

W3. RoPE ordering and positional compatibility

KVCapsule operates on the post-RoPE visual keys stored in the model's KV cache; RoPE is therefore not reapplied after reconstruction. Values follow the model's standard cache path and are not RoPE-rotated. The explicit and fused formulations preserve this ordering.

The original submission specifies that operators are indexed by visual-block token length and reused for blocks with the same processor-produced tokenization and positional layout. Appendix Q validates this fixed-operator reuse from 8 to 64 video frames. A previously unseen layout requires one-time offline calibration.

AuthorsOpenReview ↗

Continued Response to Reviewer KTkG

W4. Quest, ShadowKV, VL-Cache, and LOOK-M

Quest and ShadowKV address complementary long-context serving regimes. Quest performs query-dependent selection and loads only the top-ranked KV pages for each query, whereas ShadowKV stores a low-rank key cache, offloads values to CPU memory, and reconstructs selected sparse KV pairs during decoding. KVCapsule instead compacts the persistent visual KV cache into a reusable representation while preserving an approximate full visual-position domain, without CPU value offloading. Their released implementations target text-only model and cache interfaces; applying them to Qwen3-VL would require additional visual-block, multimodal-position, and cache-policy choices not defined by the published methods.

We therefore provide new matched-memory comparisons with LOOK-M and VisCache, two directly applicable multimodal methods with available implementations. For VL-Cache, no released official implementation supports the evaluated Qwen3-VL cache interface.

We evaluate Full KV, paper-default KVCapsule c60, LOOK-M, and VisCache on the same fixed MME (2,374 examples), Video-MME 8-frame (512), LLaVABench (60), and MMBench-Video (639) manifests. All four methods use Qwen3-VL-8B-Instruct and the same prompts, decoding settings, and dataset-specific evaluation protocol within each benchmark. The judge-based benchmarks additionally use the same locked evaluator across methods.

KVCapsule uses the paper-default c60 configuration. The LOOK-M and VisCache operating points were selected through independent memory-only calibration before quality evaluation. Persistent-memory accounting includes each method's state, so the comparison uses nearby measured reductions rather than nominal token-retention ratios.

MME reports its perception score and Video-MME reports accuracy. The two judge metrics have different scales: LLaVABench Relative Score may exceed 100, while MMBench-Video Overall is a mean 0--3 rating. All methods below share the same dataset-specific protocol and Full-KV reference. Bold marks the strongest compressed method; Full KV is the uncompressed reference.

Method10K persistent-memory reductionMME ↑\uparrow (2,374)Video-MME ↑\uparrow (512)LLaVABench ↑\uparrow (95% CI; 60)MMBench-Video ↑\uparrow (95% CI; 639)
Full KV0.00%1713.50.658111.7 [103.2, 118.9]1.65 [1.56, 1.74]
KVCapsule55.54%1722.00.65384.4 [74.7, 94.3]1.25 [1.16, 1.34]
LOOK-M54.32%1716.00.65023.2 [19.4, 28.0]1.18 [1.09, 1.27]
VisCache57.57%1711.50.64821.9 [18.6, 26.1]1.10 [1.02, 1.19]

At these nearby measured memory points, KVCapsule has the highest compressed-method point estimate on all four benchmarks. Its LLaVABench confidence interval is clearly separated from LOOK-M and VisCache. The compressed-method intervals overlap on MMBench-Video, while the smaller MME and Video-MME margins are reported as point estimates rather than significance claims.

Thank you again for your valuable feedback. As the discussion period ends on November 18, we would be grateful if you could let us know whether our rebuttal has addressed your concerns.

Thank the authors for the rebuttal. Some of my concerns are resolved. However, my core concerns about the novelty and the anomalies in Table 2 are not fully addressed. I will maintain my original rating.

Reviewer_w4LROpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper targets the visual KV cache, which dominates memory in long-context VLMs. From three observations, sequence-level redundancy in visual KV, decoding-dependent token importance, and distinct key/value redundancy structures, the authors propose KVCapsule that reconstructs visual keys from selectively retained positions and compresses visual values by sequence-level PCA, with a frozen backbone and one-time offline calibration. A fused attention path avoids materializing the reconstructed tensors. Across five backbones and image and video benchmarks, the paper reports 55.0% persistent-KV reduction and 24.0% higher decoding throughput at 10K context.

Critical strengths
  1. Sequence-dimension and K/V asymmetry framing are genuinely interesting hypothesises.
  2. The fused formulation (Algorithm 1) avoids materializing full-length tensors.
  3. Extensive experiments across 5 backbones, 6 image and 7 video benchmarks.
Critical weaknesses
  1. The observations are not VLM-specific. Asymmetric key/value structure is already established for text-only KV compression. KIVI is built on it: keys have consistent outlier channels, values do not. That distinction should be stated explicitly. Please also run the Appendix A similarity measurement on text tokens in the same models.

  2. The formalism contradicts the stated observations. The introduction and Section 2.2 claim importance varies across layers and heads. But Section 3.1 defines Mℓ\mathcal{M}\ell and θℓ={W1,ℓ,W2,ℓ}\theta\ell = \{W_{1,\ell}, W_{2,\ell}\} indexed by layer only. Appendix C then optimizes per-head masking logits. Appendix D's shapes carry an 𝐻𝑘𝑣𝐻_{𝑘𝑣} dimension. So the implementation looks per-head and the method section is written per-layer. Please fix the notation. State whether masks and reconstructors are shared across KV heads.

  3. Sequential redundancy discovery overlaps prior work. Visual-token redundancy is the explicit premise of FastV, which you use as a baseline. It also underlies the merging methods in Section 5. Calling it a discovery is not supported. Consider reframing around what is actually new: the sequence-axis low-rank characterization and the K/V asymmetry along that axis.

  4. KV cache compression is lossy, so the abstract needs an accuracy number. Right now it reports only 55.0% memory reduction and 24.0% throughput.

Questions for authors

none

2 comments
AuthorsOpenReview ↗

Response to Reviewer w4LR

We thank the reviewer for the constructive comments. We clarify these points and provide the requested modality-controlled analysis below.

W1. Relation to text-only K/V asymmetry and KIVI

Prior work including KIVI studies K/V asymmetry in text, primarily channel-wise outliers for quantization. KVCapsule addresses visual K/V redundancy along the token-sequence axis and the persistent cache created by dense visual inputs. It applies different sequence-axis operators to visual keys and values, reconstructs an approximate full-position attention domain, and supports fused execution without materializing full reconstructed K/V.

We provide a modality-controlled analysis on a fixed 500-example MME manifest. For equal-length visual and text subsets, excluding special tokens, we compute off-diagonal cosine similarity on post-RoPE cache keys and cache values per KV head and then average across heads and examples:

LayerVisual KVisual VVisual K--V gapText KText VText K--V gap
00.9940.3450.6490.9920.1050.887
100.8560.2760.5800.8330.1820.651
200.5090.1720.3380.6330.2200.413

The controlled comparison shows that K/V asymmetry occurs across modalities. KVCapsule's novelty lies in converting this structure into a complete visual-cache representation and execution design: distinct sequence-axis operators for visual keys and values, reusable per-head calibration without backbone retraining, approximate full-position reconstruction rather than permanent token deletion, and fused execution without materializing reconstructed K/V. This differs from KIVI's channel-wise quantization objective and from visual-token pruning methods that permanently restrict the retained attention domain.

Appendix A and the new table answer complementary questions: Appendix A uses a visual-only MEGABench protocol, while the table above compares equal-length visual and text subsets from the same MME examples. Both show a clear visual K--V separation.

W2. Per-layer versus per-head notation

The implementation is per layer and per KV head. The main key equations suppress the head index for compactness, while the value equations retain it. Appendix C specifies independent per-layer, per-head calibration and Appendix D carries the KV-head dimension in the memory accounting. Only the retained-length schedule rℓr_\ell is layer-wise.

We will restore the head index hh on M\mathcal M, W1W_1, W2W_2, UU, and μ\mu in the main equations and Algorithm 1 for consistency. The reported implementation and results use per-head operators.

W3. Novelty of sequential redundancy

The submission discusses visual-token pruning and merging methods and evaluates FastV, SparseVLMs, and PruMerge. KVCapsule's novelty lies in its combined representation and execution design: (i) distinct sequence-axis operators for visual keys and values, (ii) an approximate full-position attention domain rather than permanent deletion of unretained positions, (iii) reusable per-layer/per-head calibration for a frozen VLM, and (iv) algebraically fused decoding that avoids materializing reconstructed K/V. Together, these components reduce persistent visual-KV storage while keeping discarded positions approximately recoverable for later queries. We will sharpen this distinction in the contribution statement and make the FastV citation explicit.

The submitted matched-memory ablation in Appendix M further supports this design: at the same 60.2% visual-KV reduction, learned key reconstruction with value PCA achieves the strongest compressed result on five of six metrics; symmetric PCA is slightly higher only on MMMU (0.4860.486 versus 0.4800.480).

W4. Quality information in the abstract

The submitted main evaluation reports the quality--memory trade-off across multiple benchmarks. To make this result immediately visible in the abstract, we will surface one representative point:

At the default compression setting on Qwen3-VL, KVCapsule achieves 78.0% MMBench accuracy versus 80.0% for Full KV.

The abstract will state this quality result separately from the 10K memory and throughput result because they use distinct evaluation workloads.

Thank you again for your valuable feedback. As the discussion period ends on November 18, we would be grateful if you could let us know whether our rebuttal has addressed your concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.