- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces CoViST, a training-free visual token compression framework. The method tracks a "composable visual state" (incorporating original position, contribution weights, and selection metadata) for each retained token, paired with coverage-guided selection, feature composition under conservation constraints, and attention bias adjustments. The authors present both a single-stage variant (CoViST-Fixed) and a progressive multi-layer variant (CoViST-Pro), evaluating them across four VLM backbones and multiple benchmarks to demonstrate efficiency and accuracy trade-offs.
Critical strengths
-
Coherent Pipeline and Stepwise Ablations: The framework is logically constructed, accompanied by a well-structured ablation ladder that verifies the incremental contribution of each component.
-
Detailed Experimental Disclosure: The paper provides comprehensive details regarding baseline data sourcing, layer-average budget definitions, and hyperparameter sensitivity analyses.
-
Multi-Backbone and Practical Efficiency Validation: Evaluated on four standard VLM backbones, with reported efficiency metrics (e.g., latency and speedup) explicitly factoring in the overhead of constructing the visual states.
Critical weaknesses
-
Claims of Superiority Rely on Inconsistent Baseline Normalization, Marginal Margins, and Incomplete Benchmarks: The relative retention rates reported for CoViST are normalized against the paper's own uncompressed baseline, whereas competitors' numbers are pulled directly from their original publications (where reference base scores differ, e.g., on ScienceQA and MMBench). When re-evaluated under a unified reference baseline, CoViST's margin over baselines like VisionTrim becomes extremely small (less than 0.1 points at certain tight budgets) or even negative. Furthermore, the evaluation coverage is insufficient: for LLaVA-1.5-7B, while prior work typically evaluates on 10+ benchmarks, this paper tests on only 7, making the claims of consistent state-of-the-art leadership across budgets less solid.
-
Limited Novelty of Core Components and Confounded Ablations: Each individual component closely resembles established prior art (e.g., coverage selection mirrors MMTok/SCOPE, confidence-gated merging follows CaRe, log-count attention weights build on ToMe, and RoPE distance factors align with RESTORE). While the main contribution lies in their combination, the paper lacks module-swapping ablation studies against these published alternatives within the same pipeline, making it hard to isolate the exact source of performance gains.
-
Unfair Budget Comparison and Questionable Utility of the Progressive Variant: CoViST-Pro retains significantly more tokens in earlier layers. Although its layer-average token budget matches Fixed, its computational cost (TFLOPs) is higher, making direct comparisons with fixed-budget baselines at the same "average token count" somewhat unfair. More importantly, CoViST-Pro yields marginal accuracy gains over Fixed while offering an unfavorable speedup-to-performance trade-off. For example, CoViST-Fixed at achieves comparable speedups to CoViST-Pro at , yet significantly outperforms it in accuracy, raising doubts about the practical value of the progressive compression design.
Questions for authors
-
Unified Baseline Normalization and Benchmark Expansion: Can you report benchmark performance using a unified reference uncompressed baseline across all methods, and expand the evaluation to a broader benchmark set (matching the 10+ datasets standard in prior work) to ensure a solid comparison?
-
Component-Level Swap Ablations: Can you conduct ablation experiments swapping individual components with their published counterparts (e.g., ToMe's count weighting, RESTORE's shared gain) under the exact same pipeline?
-
Fair Efficiency Comparisons: How do CoViST-Fixed and CoViST-Pro compare against baselines like VisionZip, ZOO-Prune, and VisionTrim when evaluated at equal TFLOPs and actual hardware latency, and what is the practical justification for CoViST-Pro over a higher-budget CoViST-Fixed?
6 comments
(W1&Q1 part 2)
For transparency, the table below lists the uncompressed references used for CoViST and all reproduced baselines in our environment, alongside the references reported in the competing methods’ papers. The original LLaVA-1.5 results are included for context. Among the original seven benchmarks, the two largest difference in percentage-valued reference scores between CoViST and the compared token compression methods occurs on VQAv2 and MMB-CN: 76.7 vs. 78.5, and 55.5 vs. 58.3. Differences also exist between the original LLaVA-1.5 results and the references reported in token compression papers. These observations reinforce the importance of preserving the correspondence between each compressed result and its uncompressed reference, as well as providing controlled reproductions.
| Source | GQA | MMB | MME | POPE | SQA | TextVQA | VQAv2 | MMB-CN | SEED-I | MMMU | VizWiz |
|---|---|---|---|---|---|---|---|---|---|---|---|
| LLaVA-1.5 | 62.0 | 64.3 | — | 85.9 | 66.8 | 58.2 | 78.5 | 58.3 | 66.1 | — | 50.0 |
| CoViST | 62.0 | 64.0 | 1867 | 85.8 | 69.5 | 58.2 | 76.7 | 55.5 | 66.2 | 36.0 | 50.5 |
| VisionTrim | 61.9 | 64.7 | 1862 | 85.9 | 69.5 | 58.2 | 78.5 | — | — | — | 50.1 |
| DeSAP | 61.9 | 64.7 | 1862 | 85.9 | 69.5 | 58.2 | 78.5 | — | — | — | 50.0 |
| RESTORE+HoloV | 61.9 | 64.6 | 1862 | 85.9 | 69.5 | 58.2 | 78.5 | — | — | — | — |
| MMTok | 61.9 | 64.7 | 1862 | 85.9 | 69.5 | 58.2 | 78.5 | — | 66.1 | 36.3 | — |
| ZOO-Prune | 61.9 | 64.7 | 1862 | 85.9 | 69.5 | 58.2 | 78.5 | — | — | 36.3 | — |
| VisionZip | 61.9 | 64.7 | 1862 | 85.9 | 69.5 | 58.2 | 78.5 | — | — | 36.3 | 50.0 |
These results are added to appendix E.2 of the revised paper.
(W2&Q2)
Thank you for this suggestion. Our main contribution is the composable visual state and the joint formulation of CSI, CSC, and SCA around this representation. CoViST explicitly maintains representative content, original positions, effective contribution weights, and selection metadata. Its composition rule produces another state of the same form after further compression, and decoder attention uses the resulting contribution and positional information. This formulation specifies how compressed visual information is maintained across reductions and used by the decoder, providing a common basis for fixed and progressive compression.
To directly assess the individual design choices, we conducted component swapping experiments on LLaVA-1.5-7B at 192/128/64 tokens. Each experiment changes one component within the CoViST-Fixed pipeline while retaining the token budget and the remaining component rules:
- ToMe's count weighting: replacing the effective contribution weights with unclipped patch counts, retaining the correspondence in CSC, content composition, original positions, distance factor, and per-head gains.
- MMTok coverage selection: replacing the representative selection with the MMTok selector on the same patch candidate set, retaining CoViST's composition and attention rules. Note that we use the official MMTok code here.
- RESTORE's shared gain: set the per-head gains in SCA to 1.0 across all layers and heads, while retaining the effective contribution weights, original positions, and distance factor.
The table below compares these substitutions with the original CoViST-Fixed:
| K | Configuration | GQA | MMB | MME | POPE | SQA | TextVQA | R |
|---|---|---|---|---|---|---|---|---|
| 192 | CoViST-Fixed | 61.3 | 64.3 | 1901 | 86.2 | 69.1 | 57.6 | 99.96 |
| 192 | w/ ToMe's count weighting | 61.3 | 65.0 | 1873 | 86.3 | 69.4 | 57.5 | 100.00 |
| 192 | w/ MMTok coverage selection | 61.1 | 64.1 | 1830 | 86.1 | 69.0 | 58.0 | 99.33 |
| 192 | w/ RESTORE's shared gain | 61.5 | 64.9 | 1863 | 86.4 | 69.1 | 57.3 | 99.84 |
| 128 | CoViST-Fixed | 61.3 | 64.1 | 1867 | 86.3 | 69.0 | 57.1 | 99.49 |
| 128 | w/ ToMe's count weighting | 61.3 | 64.1 | 1858 | 86.3 | 68.9 | 57.0 | 99.37 |
| 128 | w/ MMTok coverage selection | 60.7 | 63.8 | 1806 | 85.7 | 69.0 | 57.6 | 98.75 |
| 128 | w/ RESTORE's shared gain | 61.2 | 64.9 | 1833 | 86.6 | 68.3 | 57.0 | 99.25 |
| 64 | CoViST-Fixed | 60.2 | 63.1 | 1819 | 85.8 | 68.5 | 56.4 | 98.10 |
| 64 | w/ ToMe's count weighting | 60.2 | 63.2 | 1810 | 85.7 | 68.3 | 56.2 | 97.93 |
| 64 | w/ MMTok coverage selection | 59.9 | 63.1 | 1805 | 85.2 | 68.8 | 56.1 | 97.77 |
| 64 | w/ RESTORE's shared gain | 60.3 | 64.5 | 1817 | 86.1 | 68.6 | 56.2 | 98.50 |
The swaps reveal consistent performance gains from our selection design: replacing it with MMTok coverage selection reduces performance retention at all three budgets. ToMe's count weighting gives slightly higher retention at 192 tokens and slightly lower retention at 128 and 64 tokens, suggesting modest benefits from our effective contribution weights at the smaller budgets. The RESTORE's' shared gain reduces retention at 192 and 128 tokens but improves it at 64 tokens. These comparisons provide direct evidence about the roles of the specific method designs within CoViST, complementing the comparisons with complete baseline methods.
We want to emphasize again that our main contribution is the composable visual state and the composition and attention rules that operate consistently on that state. This formulation connects selection, contribution transfer, and decoder attention through an explicit representation that remains usable after further reductions. The comparisons with other SOTA methods demonstrate the effectiveness of this joint design through CoViST's higher overall performance retention. All swap variants continue to operate within this framework and retain the remaining CoViST mechanisms, so their strong performance is obtained with substantial parts of our formulation still in place. We have included these experiments in the appendix D.4 of revised paper.
(W1&Q1 part 1)
Thank you for highlighting these concerns. We have conducted controlled evaluations of Six competitive and representative methods—VisionTrim, DeSAP, MMTok, VisionZip, ZOO-Prune, and RESTORE+HoloV—under a unified evaluation harness. Specifically, we use the available official code for VisionTrim, MMTok, VisionZip, ZOO-Prune, and RESTORE+HoloV, while implement DeSAP by ourselves based on its paper. We also expand the evaluation from seven to eleven benchmarks by adding MMBench-CN (MMB-CN), SEEDBench-Image (SEED-I), MMMU, and VizWiz.
In our response to Reviewer B4HU (W1/Q1), we present both the paper-reported results and the results obtained in our environment. CoViST and all reproduced baselines use exactly the same uncompressed reference in the controlled comparison. We denote the average performance retention over the original seven benchmarks and the expanded eleven benchmarks as R7 and R11, respectively. The following table summarizes R7 and R11 across all three token budgets, please refer to our response to Reviewer B4HU (W1/Q1) for more detailed results on every benchmark.
At 192/128/64 tokens, RESTORE's published R7 values are 98.8%/98.2%/96.5%, while our reproduction gives slightly better 98.9%/98.5%/96.7%, making it the strongest baseline. For ZOO-Prune, the published values are 98.3%/97.6%/95.2%, and ours are 98.3%/97.1%/95.0%. For DeSAP, the corresponding values are 99.1%/98.3%/96.8% in the paper and 98.1%/97.3%/94.7% in our implementation. Our reproductions of MMTok and VisionZip also achieve comparable or slightly higher R7 values than their reported results. However, for VisionTrim, published R7 values are 99.7%/99.0%/98.0%, while our reproduction gives 97.1%/94.9%/90.1%. It is the most competitive method with paper-reported results while the weakest one with reproduced results using it official code. Most importantly, CoViST still ranks first and second when compared with either paper-reported or reproduced results using R7, and with all six reproduced methods using R11.
Regarding normalization, performance retention measures the performance preserved by compression relative to the corresponding uncompressed model under the same experimental setting. Each published compressed score should therefore be paired with the uncompressed reference from that source’s experimental setting. Retaining published compressed scores while replacing their denominators with a reference obtained under another protocol would confound compression retention with differences in the reference evaluation and would not constitute a controlled comparison. The results normalized within each source provide context, while our common-harness reproductions directly address the requested comparison under a unified reference.