- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper first investigates the redundancy of vision tokens in VLMs. It then proposes KVCapsule, which compresses keys and values asymmetrically. Keys are selected with a learned, fixed sequence mask and reconstructed to the original sequence length using a two-layer MLP. Values are centered and projected through a learned sequence-dimensional PCA basis, then reconstructed by linear re-projection.
Critical strengths
- Substantial investigative experiments. The paper includes a substantial amount of experiments to first investigate and quantify the redundancy of vision tokens in VLMs.
- Following the above, I believe the method is well motivated and grounded on prior investigative experiments. Specifically, the asymmetrical compression of keys and values is grounded and novel.
Critical weaknesses
- While the investigative experiments provide useful insights in understanding the redundancy of vision tokens, the underlying mechanism behind the observed key value asymmetry remains insufficiently explored. Since keys and values are derived from the same underlying token representations, why do keys exhibit substantially higher inter-token similarity (particularly in earlier layers), while values remain more token-specific (Fig. 4)? A deeper explanation of this phenomenon would strengthen the motivation behind the asymmetric compression.
- Missing details in method. The paper provides calibration procedures in Appendix C, but the provenance of the calibration dataset and its relationship to the evaluation datasets remain insufficiently specified. In particular, are the 500 calibration samples strictly disjoint from all downstream evaluation samples? Moreover, how sensitive are the learned operators to the calibration distribution? Since the method relies on fixed, learned reconstruction operators, the authors should investigate and provide evidence of generalization across unseen data distributions.
- The efficiency evaluation remains insufficiently characterized. In particular, Fig. 5(c) shows that attention latency under Full KV increases with input length, whereas KVCapsule exhibits a slightly decreasing trend despite maintaining a fixed compression configuration. This behavior is counterintuitive and warrants further explanation. The authors should clarify the workload composition, how they were measured, and sources of latency variation. Preferably, the authors should also report variability across repeated measurements.
- The method introduces additional score-space computations, including an term per visual block (Sec 4.3: since d_b = n/4, is essential ) . The authors should provide a more detailed analysis of this overhead and clarify how reconstruction costs scale with visual-token length and different tokenization configurations. In particular, it would be useful to identify in which configs the reconstruction overhead outweighs the benefits of reduced KV-memory traffic.
- Need for in-depth analysis of reconstruction. App. G reports attention pattern cosine similarities of 0.58, 0.76, and 0.60 for K-only, V-only, and joint K+V compression. However, I think the experimental protocol and interpretation of these values are insufficiently explained. In particular, a similarity of 0.60 suggests substantial deviations from full-KV attention patterns, despite the relatively small downstream performance differences reported in Tables 2 and 3. Furthermore, App R reports a PPL increase of 2.23x relative to full KV, raising additional questions about output quality. The authors should investigate these discrepancies and provide a more comprehensive analysis of how reconstruction errors propagate to attention outputs and final generation quality.
- While Tables 2 and 3 provide extensive downstream benchmark comparisons, corresponding efficiency measurements are largely deferred to the appendices. Although Appendix K and P provide useful comparisons, the multi-budget evaluation primarily considers PruMerge. Given the efficiency-focused nature of the paper, a more comprehensive comparison against alternative compression methods under matched memory budgets, reporting both downstream performance and inference efficiency, would strengthen the experimental evaluation.
Overall, the paper produces strong investigative evidence and a well motivated approach. However, several important aspects of the methodology and experimental evaluation remain insufficiently explained.
Questions for authors
See weaknesses.
3 comments
Continued Response to Reviewer Xg2c
W4. Reconstruction overhead and scaling
Section 4.3 derives the fused-path complexity, including the score-reconstruction term when , and reports a persistent-memory break-even near . Appendix Q specifies that calibrated operators are shared across repeated blocks with the same visual tokenization.
For batch size , number of visual blocks , uncompressed and retained lengths and , head width , and element size , the persistent visual-cache footprint scales approximately as
where is shared rather than replicated across blocks. This gives the memory break-even: shared operator state matters most at small , while the reduction dominates as context or batch grows. Runtime also includes the reconstruction arithmetic identified above; W3 measures when the reduced traffic outweighs that execution cost.
An idealized count based on BF16 cache reads and GQA reuse gives approximately 4.0 FLOP/byte for Full KV and 7.8 FLOP/byte for KVCapsule, both below the MI250X BF16/HBM ridge point of approximately 120 FLOP/byte. Both paths are therefore memory-bound in this regime; KVCapsule reduces visual-cache traffic and raises the attainable memory-roof bound. “Fused” refers to algebraically avoiding materialization of reconstructed full-length K/V. The reported implementation uses standard operations without a custom KVCapsule kernel.
We repeat the W3 protocol using independently calibrated c60 operators for
468-token visual blocks:
| Visual tokens/block | Batch | Processed context | Full KV TPOT (ms) | KVCapsule TPOT (ms) | Throughput change | Persistent-memory reduction |
|---|---|---|---|---|---|---|
| 468 | 1 | 7,782 | 54.481 [54.479, 54.483] | 49.842 [49.760, 49.927] | +9.3% | 46.23% |
| 468 | 1 | 10,132 | 61.367 [61.363, 61.373] | 49.525 [49.378, 49.694] | +23.9% | 49.70% |
| 468 | 1 | 15,302 | 76.542 [76.537, 76.552] | 49.822 [49.696, 49.957] | +53.7% | 53.57% |
| 468 | 4 | 7,782 | 74.399 [74.394, 74.404] | 52.661 [52.657, 52.665] | +41.3% | 51.36% |
| 468 | 4 | 10,132 | 87.300 [87.293, 87.308] | 56.562 [56.553, 56.577] | +54.3% | 53.64% |
| 468 | 4 | 15,302 | 116.183 [116.173, 116.195] | 65.209 [65.203, 65.219] | +78.2% | 56.19% |
KVCapsule is faster in all six settings with non-overlapping TPOT confidence intervals, and its advantage increases with context and batch size. Together with W3, this demonstrates the same scaling behavior for independently calibrated 117- and 468-token visual blocks. The larger method-state term for 468-token operators is included in every reported memory measurement.
W5. Reconstruction fidelity, attention similarity, and perplexity
Appendix R reports output-level fidelity on a fixed deterministic 60-example LLaVABench subset using attention-output SNR, decoding-step logit KL, top-1 agreement, and sequence-level perplexity.
The reported relative PPL is computed by scoring the same 20-token Full-KV continuation under Full KV and KVCapsule. Across 60 examples and 1,200 tokens, the absolute PPL values are and , respectively. It therefore measures fidelity to Full KV's exact continuation, whereas Tables 2 and 3 evaluate downstream answer quality.
On this fidelity diagnostic, KVCapsule achieves 18--20 dB attention-output SNR; logit KL changes from to and top-1 agreement from to over decoding steps 1--20. K-only and joint compression also have nearly identical PPL ( versus ), while joint compression increases KV reduction from 30.1% to 60.2%. Thus, value compression provides the additional memory reduction without further degradation on this diagnostic.
W6. Matched-memory comparisons
Appendices K and P provide complementary comparisons using measured persistent memory. Appendix K reports memory and throughput for Full KV, FastV, PruMerge, KVCapsule, and SparseVLMs under the same 10K setting. Appendix P compares KVCapsule and PruMerge at approximately 40%, 60%, and 70% measured reduction, including method-specific state and downstream quality. The evaluated SparseVLMs implementation sparsifies attention without compacting stored K/V, so it remains in the 10K efficiency comparison but not the multi-budget cache-compression curve.
The controlled LOOK-M/VisCache evaluation above further compares KVCapsule, LOOK-M, and VisCache at nearby measured reductions of 55.54%, 54.32%, and 57.57%, respectively, across four image and video benchmarks. Together, these experiments provide efficiency, multi-budget, and downstream-quality comparisons using measured memory rather than nominal token-retention ratios.
Thank you again for your valuable feedback. As the discussion period ends on November 18, we would be grateful if you could let us know whether our rebuttal has addressed your concerns.
Response to Reviewer Xg2c
We thank the reviewer for the questions and feedback.
W1. Why keys and values exhibit different inter-token structure
Under the visual-only MEGABench protocol in Appendix A, visual-key inter-token cosine similarity is 0.99/0.87/0.66 at layers 0/10/20, compared with 0.35/0.40/0.30 for visual values. Across the same layers, values require 113--114 of 128 dimensions to preserve 95% variance and 125 dimensions to preserve 99%, showing that their information remains dense across feature channels.
At the same 60.2% visual-KV reduction, learned key reconstruction plus value PCA gives the best compressed result on five of six metrics; symmetric PCA is slightly higher only on MMMU (0.486 versus 0.480). Together, the representation measurements and matched-memory ablation show that the structural difference translates into stronger downstream performance at a fixed compression level.
W2. Calibration data, reuse, and source sensitivity
Appendix C specifies that the calibration set contains 500 single-image examples, that operators are fitted independently for each layer and KV head, and that the resulting bank is frozen and reused across image and video evaluations. The submitted bank is fitted on uncompressed visual K/V activations from 500 MEGABench images; no downstream labels, answers, scores, or evaluation examples are used. Appendix Q further evaluates reuse of a fixed bank across inputs containing 8 to 64 video frames. We will consolidate these details next to the main experimental setup.
To directly test sensitivity to the calibration source, we separately fitted
c60banks using 400 MEGABench images and 400 MLVU videos while keeping the model, fitting procedure, downstream manifests, decoding, and evaluator fixed:The image- and video-derived banks differ by only 0.7 on LLaVABench and 0.06 on MMBench-Video. Thus, in both evaluations, KVCapsule transfers between image- and video-derived calibration banks without benchmark-specific or per-input fitting.
W3. Runtime crossover under repeated measurement
The submission reports an attention-path profile at 9K--15K and a 24.0% end-to-end throughput gain at 10K. We further measure whole-model decoding across context lengths and batch sizes using 117-token visual blocks and the same
c60configuration. Each cell uses 20 warmup iterations, 100 synchronized timed measurements, and 128 greedy output tokens on one AMD MI250X. We report mean time per output token (TPOT) with bootstrap 95% confidence intervals over the timed measurements.For 117-token visual blocks, KVCapsule crosses Full KV at 10K for batch 1 and 5K for batch 4, with non-overlapping confidence intervals at all five beneficial points. At batch 1 and 5K, reduced KV traffic does not yet offset the additional reconstruction overhead, producing a 4.0% throughput reduction while still reducing persistent memory by 56.3%.
The small batch-1 TPOT change from 50.17 ms at 10K to 48.42 ms at 15K is reproducible under this protocol. Standard-operator GPU latency can vary with tensor shape, so this local fluctuation does not imply decreasing algorithmic cost. The relevant result is the measured crossover and the substantially flatter long-context scaling: Full-KV TPOT continues to increase with context, whereas KVCapsule's reduced KV traffic increasingly offsets its additional reconstruction overhead.
Memory reduction varies across workloads because text KV remains uncompressed and fixed method state is amortized over different context and batch sizes. Each percentage is therefore measured against its corresponding Full-KV reference.