acceptodds
Under review as a conference paper at ICLR 2027

CoViST: Visual Token Compression via Composable States

Abstract

Visual token compression lowers the inference cost of vision–language models by representing images with fewer tokens. However, most existing methods compress visual tokens to a reduced set, leaving the amount of visual evidence represented by each token and its original spatial context implicit. Therefore, the compressed representation does not explicitly encode how much visual information each representative carries or where it lies in the original image. This limitation arises even after a single reduction and becomes more pronounced when compression is repeated across decoder layers. To address this issue, we propose CoViST, a training-free framework that represents a compressed image as a composable visual state. Specifically, the state combines representative features with original positions, effective contribution weights, and reusable selection metadata. CoViST constructs this state through coverage-guided selection and conservation-based contribution composition, and explicitly incorporates its contribution and positional information into decoder attention. Each component of the state retains its interpretation under successive reductions, enabling the same formulation to support both fixed compression before prefill and progressive compression within the decoder. Experimental results on seven LLaVA-1.5-7B benchmarks show that CoViST-Fixed retains 99.9%, 99.5%, and 98.1% of uncompressed performance at 192, 128, and 64 tokens, respectively, and CoViST-Pro retains 99.8%, 99.9%, and 99.1% at the corresponding layer-average budgets, outperforming state-of-the-art methods under their respective budget settings. Code will be released publicly.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
visual token compression, efficient vision-language model

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_f7ymOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces CoViST, a training-free visual token compression framework. The method tracks a "composable visual state" (incorporating original position, contribution weights, and selection metadata) for each retained token, paired with coverage-guided selection, feature composition under conservation constraints, and attention bias adjustments. The authors present both a single-stage variant (CoViST-Fixed) and a progressive multi-layer variant (CoViST-Pro), evaluating them across four VLM backbones and multiple benchmarks to demonstrate efficiency and accuracy trade-offs.

Critical strengths
  • Coherent Pipeline and Stepwise Ablations: The framework is logically constructed, accompanied by a well-structured ablation ladder that verifies the incremental contribution of each component.

  • Detailed Experimental Disclosure: The paper provides comprehensive details regarding baseline data sourcing, layer-average budget definitions, and hyperparameter sensitivity analyses.

  • Multi-Backbone and Practical Efficiency Validation: Evaluated on four standard VLM backbones, with reported efficiency metrics (e.g., latency and speedup) explicitly factoring in the overhead of constructing the visual states.

Critical weaknesses
  • Claims of Superiority Rely on Inconsistent Baseline Normalization, Marginal Margins, and Incomplete Benchmarks: The relative retention rates reported for CoViST are normalized against the paper's own uncompressed baseline, whereas competitors' numbers are pulled directly from their original publications (where reference base scores differ, e.g., on ScienceQA and MMBench). When re-evaluated under a unified reference baseline, CoViST's margin over baselines like VisionTrim becomes extremely small (less than 0.1 points at certain tight budgets) or even negative. Furthermore, the evaluation coverage is insufficient: for LLaVA-1.5-7B, while prior work typically evaluates on 10+ benchmarks, this paper tests on only 7, making the claims of consistent state-of-the-art leadership across budgets less solid.

  • Limited Novelty of Core Components and Confounded Ablations: Each individual component closely resembles established prior art (e.g., coverage selection mirrors MMTok/SCOPE, confidence-gated merging follows CaRe, log-count attention weights build on ToMe, and RoPE distance factors align with RESTORE). While the main contribution lies in their combination, the paper lacks module-swapping ablation studies against these published alternatives within the same pipeline, making it hard to isolate the exact source of performance gains.

  • Unfair Budget Comparison and Questionable Utility of the Progressive Variant: CoViST-Pro retains significantly more tokens in earlier layers. Although its layer-average token budget matches Fixed, its computational cost (TFLOPs) is higher, making direct comparisons with fixed-budget baselines at the same "average token count" somewhat unfair. More importantly, CoViST-Pro yields marginal accuracy gains over Fixed while offering an unfavorable speedup-to-performance trade-off. For example, CoViST-Fixed at K=320K=320 achieves comparable speedups to CoViST-Pro at K=160K=160, yet significantly outperforms it in accuracy, raising doubts about the practical value of the progressive compression design.

Questions for authors
  • Unified Baseline Normalization and Benchmark Expansion: Can you report benchmark performance using a unified reference uncompressed baseline across all methods, and expand the evaluation to a broader benchmark set (matching the 10+ datasets standard in prior work) to ensure a solid comparison?

  • Component-Level Swap Ablations: Can you conduct ablation experiments swapping individual components with their published counterparts (e.g., ToMe's count weighting, RESTORE's shared gain) under the exact same pipeline?

  • Fair Efficiency Comparisons: How do CoViST-Fixed and CoViST-Pro compare against baselines like VisionZip, ZOO-Prune, and VisionTrim when evaluated at equal TFLOPs and actual hardware latency, and what is the practical justification for CoViST-Pro over a higher-budget CoViST-Fixed?

6 comments

(W1&Q1 part 1)

Thank you for highlighting these concerns. We have conducted controlled evaluations of Six competitive and representative methods—VisionTrim, DeSAP, MMTok, VisionZip, ZOO-Prune, and RESTORE+HoloV—under a unified evaluation harness. Specifically, we use the available official code for VisionTrim, MMTok, VisionZip, ZOO-Prune, and RESTORE+HoloV, while implement DeSAP by ourselves based on its paper. We also expand the evaluation from seven to eleven benchmarks by adding MMBench-CN (MMB-CN), SEEDBench-Image (SEED-I), MMMU, and VizWiz.

In our response to Reviewer B4HU (W1/Q1), we present both the paper-reported results and the results obtained in our environment. CoViST and all reproduced baselines use exactly the same uncompressed reference in the controlled comparison. We denote the average performance retention over the original seven benchmarks and the expanded eleven benchmarks as R7 and R11, respectively. The following table summarizes R7 and R11 across all three token budgets, please refer to our response to Reviewer B4HU (W1/Q1) for more detailed results on every benchmark.

MethodSource192 R7 (%)192 R11 (%)128 R7 (%)128 R11 (%)64 R7 (%)64 R11 (%)
CoViST-ProLocal99.899.499.999.299.198.4
CoViST-FixedLocal99.999.399.598.698.197.5
RESTORE+HoloVReproduced98.998.898.598.396.796.6
RESTORE+HoloVPaper98.8—98.2—96.5—
ZOO-PruneReproduced98.398.697.197.995.095.5
ZOO-PrunePaper98.3—97.6—95.2—
MMTokReproduced98.498.597.697.796.096.2
MMTokPaper98.2—97.5—96.0—
DeSAPReproduced98.198.497.397.894.795.2
DeSAPPaper99.1—98.3—96.8—
VisionZipReproduced97.798.096.497.392.893.9
VisionZipPaper97.7—96.2—92.7—
VisionTrimReproduced97.197.794.996.190.191.9
VisionTrimPaper99.7—99.0—98.0—

At 192/128/64 tokens, RESTORE's published R7 values are 98.8%/98.2%/96.5%, while our reproduction gives slightly better 98.9%/98.5%/96.7%, making it the strongest baseline. For ZOO-Prune, the published values are 98.3%/97.6%/95.2%, and ours are 98.3%/97.1%/95.0%. For DeSAP, the corresponding values are 99.1%/98.3%/96.8% in the paper and 98.1%/97.3%/94.7% in our implementation. Our reproductions of MMTok and VisionZip also achieve comparable or slightly higher R7 values than their reported results. However, for VisionTrim, published R7 values are 99.7%/99.0%/98.0%, while our reproduction gives 97.1%/94.9%/90.1%. It is the most competitive method with paper-reported results while the weakest one with reproduced results using it official code. Most importantly, CoViST still ranks first and second when compared with either paper-reported or reproduced results using R7, and with all six reproduced methods using R11.

Regarding normalization, performance retention measures the performance preserved by compression relative to the corresponding uncompressed model under the same experimental setting. Each published compressed score should therefore be paired with the uncompressed reference from that source’s experimental setting. Retaining published compressed scores while replacing their denominators with a reference obtained under another protocol would confound compression retention with differences in the reference evaluation and would not constitute a controlled comparison. The results normalized within each source provide context, while our common-harness reproductions directly address the requested comparison under a unified reference.

(W1&Q1 part 2)

For transparency, the table below lists the uncompressed references used for CoViST and all reproduced baselines in our environment, alongside the references reported in the competing methods’ papers. The original LLaVA-1.5 results are included for context. Among the original seven benchmarks, the two largest difference in percentage-valued reference scores between CoViST and the compared token compression methods occurs on VQAv2 and MMB-CN: 76.7 vs. 78.5, and 55.5 vs. 58.3. Differences also exist between the original LLaVA-1.5 results and the references reported in token compression papers. These observations reinforce the importance of preserving the correspondence between each compressed result and its uncompressed reference, as well as providing controlled reproductions.

SourceGQAMMBMMEPOPESQATextVQAVQAv2MMB-CNSEED-IMMMUVizWiz
LLaVA-1.562.064.3—85.966.858.278.558.366.1—50.0
CoViST62.064.0186785.869.558.276.755.566.236.050.5
VisionTrim61.964.7186285.969.558.278.5———50.1
DeSAP61.964.7186285.969.558.278.5———50.0
RESTORE+HoloV61.964.6186285.969.558.278.5————
MMTok61.964.7186285.969.558.278.5—66.136.3—
ZOO-Prune61.964.7186285.969.558.278.5——36.3—
VisionZip61.964.7186285.969.558.278.5——36.350.0

These results are added to appendix E.2 of the revised paper.

(W2&Q2)

Thank you for this suggestion. Our main contribution is the composable visual state and the joint formulation of CSI, CSC, and SCA around this representation. CoViST explicitly maintains representative content, original positions, effective contribution weights, and selection metadata. Its composition rule produces another state of the same form after further compression, and decoder attention uses the resulting contribution and positional information. This formulation specifies how compressed visual information is maintained across reductions and used by the decoder, providing a common basis for fixed and progressive compression.

To directly assess the individual design choices, we conducted component swapping experiments on LLaVA-1.5-7B at 192/128/64 tokens. Each experiment changes one component within the CoViST-Fixed pipeline while retaining the token budget and the remaining component rules:

  • ToMe's count weighting: replacing the effective contribution weights with unclipped patch counts, retaining the correspondence in CSC, content composition, original positions, distance factor, and per-head gains.
  • MMTok coverage selection: replacing the representative selection with the MMTok selector on the same patch candidate set, retaining CoViST's composition and attention rules. Note that we use the official MMTok code here.
  • RESTORE's shared gain: set the per-head gains in SCA to 1.0 across all layers and heads, while retaining the effective contribution weights, original positions, and distance factor.

The table below compares these substitutions with the original CoViST-Fixed:

KConfigurationGQAMMBMMEPOPESQATextVQAR
192CoViST-Fixed61.364.3190186.269.157.699.96
192w/ ToMe's count weighting61.365.0187386.369.457.5100.00
192w/ MMTok coverage selection61.164.1183086.169.058.099.33
192w/ RESTORE's shared gain61.564.9186386.469.157.399.84
128CoViST-Fixed61.364.1186786.369.057.199.49
128w/ ToMe's count weighting61.364.1185886.368.957.099.37
128w/ MMTok coverage selection60.763.8180685.769.057.698.75
128w/ RESTORE's shared gain61.264.9183386.668.357.099.25
64CoViST-Fixed60.263.1181985.868.556.498.10
64w/ ToMe's count weighting60.263.2181085.768.356.297.93
64w/ MMTok coverage selection59.963.1180585.268.856.197.77
64w/ RESTORE's shared gain60.364.5181786.168.656.298.50

The swaps reveal consistent performance gains from our selection design: replacing it with MMTok coverage selection reduces performance retention at all three budgets. ToMe's count weighting gives slightly higher retention at 192 tokens and slightly lower retention at 128 and 64 tokens, suggesting modest benefits from our effective contribution weights at the smaller budgets. The RESTORE's' shared gain reduces retention at 192 and 128 tokens but improves it at 64 tokens. These comparisons provide direct evidence about the roles of the specific method designs within CoViST, complementing the comparisons with complete baseline methods.

We want to emphasize again that our main contribution is the composable visual state and the composition and attention rules that operate consistently on that state. This formulation connects selection, contribution transfer, and decoder attention through an explicit representation that remains usable after further reductions. The comparisons with other SOTA methods demonstrate the effectiveness of this joint design through CoViST's higher overall performance retention. All swap variants continue to operate within this framework and retain the remaining CoViST mechanisms, so their strong performance is obtained with substantial parts of our formulation still in place. We have included these experiments in the appendix D.4 of revised paper.

Reviewer_FUtWOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces CoViST, a training-free visual token compression method that represents compressed images as composable visual states. Each token incorporates information from some discarded tokens along with their contribution weights, allowing the state components to retain their meaning across successive reduction steps. CoViST outperforms the compared baselines under both Fixed and Pro settings, supported by extensive and convincing experiments.

Critical strengths
  1. The paper is well written, and the role of each component is clearly explained.
  2. Extensive experiments demonstrate clear advantages across different backbones and compression ratios. Thorough ablation studies establish the contribution of each component.
Critical weaknesses
  1. The proposed mechanism is interesting, but a visual illustration would substantially improve clarity. In particular, the authors could show which tokens are retained and how information from discarded tokens is assigned to the retained tokens.
  2. The paper would benefit from representative failure cases and an analysis of the method’s limitations.
Questions for authors

See the weaknesses.

2 comments

(W1)

Thanks for your constructive suggestion. We have added visualizations of CoViST-Fixed on LLaVA-1.5-7B at 64 and 192 tokens in Figure 4-5 in the revised paper, with additional examples in Appendix H (Figure 6-8). The figures show which tokens CSI retains and how CSC assigns discarded tokens to retained representatives. We further illustrate how SCA incorporates the composed visual state into decoder attention in these figures. Section 4.6 discusses the complementary roles of the three components. As figures cannot be added here, please refer to the revised paper to check them.

(W2)

Thanks for highlighting this concern. We have added representative failure cases and a discussion of limitations in Section 4.6 (Figure 5) in the revised paper, with additional examples in Appendix I (Figures 7 and 8). The visualizations present the questions, correct answers, model outputs, and the corresponding CSI, CSC, and SCA results. These examples cover three types of failures: errors resolved by increasing the budget from 64 to 192 tokens, errors that persist at both budgets, and predictions that become incorrect at the larger budget. They illustrate limitations in preserving the visual details and local structure required for fine-grained recognition, small-object recognition, spatial reasoning, and counting. Although CSC composes effective contributions and SCA incorporates them into attention, these operations cannot always guarantee preservation of every task-relevant visual feature. As figures cannot be added here, please refer to the revised paper to check them.

These qualitative findings also complement our response on category-level errors at 64 tokens regarding Reviewer B4HU's W3&Q3. In particular, the numerical-calculation, counting, and spatial-relation examples provide qualitative context for the decreases observed in MME Numerical Calculation and Count, and MMB Relation Reasoning for CoViST-Fixed. The small-object example further illustrates a limitation that is not separately captured by the existing category labels.

Thank you for the response. The clarifications effectively address my concerns and strengthen the paper. I maintain my positive assessment.

Reviewer_B4HUOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper argues that training-free visual token compression makes the evidence carried by each retained token implicit: neither how much visual information a representative stands for nor where it came from is encoded, and this loss compounds under progressive (multi-layer) compression. It proposes to represent the compressed image as a composable visual state that carries representative features, original positions, effective contribution weights and reusable selection metadata. Three modules build and maintain it: Coverage-Guided State Initialization (CSI) selects representatives by coverage of visual and instruction-relevant evidence; Coupled State Composition (CSC) transfers the contribution of removed tokens to their representatives under a per-image conservation constraint, inherits weights across reductions and applies a confidence-gated feature update; State-Conditioned Attention (SCA) injects the weights and original positions into every decoder layer through closed-form head gains derived from the frozen projections. The same formulation supports CoViST-Fixed (one reduction to K tokens before prefill) and CoViST-Pro (progressive reduction at two intermediate layers). Experiments cover seven benchmarks on LLaVA-1.5-7B, plus LLaVA-1.5-13B, LLaVA-NeXT-7B and Qwen2.5-VL-7B, three budgets each, and an efficiency study (TTFT, KV cache, decoder prefill FLOPs).

Critical strengths

S1. A conceptual contribution that fixes a real defect of progressive compression. Existing progressive methods (e.g. PyramidDrop) reduce twice, but the second reduction operates on representations that already aggregate several original tokens with no record of what the first stage established. Making the state closed under reduction — content, position, evidence weight, selection metadata — is the right abstraction, and the paper shows it supports both fixed and progressive execution with one formulation. The ablation of state components (Table 6) is consistent with each component mattering.

S2. Breadth and honesty of the empirical protocol. Four models, seven benchmarks, three budgets each, with the same ranking across budgets and architectures (Tables 1–4, 10–11); a real efficiency study on LLaVA-NeXT (TTFT 264 → 179/126/103 ms, decoder prefill FLOPs 43.4T → 10.0/5.7/3.5T, KV cache 1495 → 375/215/135 MiB, generation speedup 1.39×/1.89×/2.22×); and an explicit statement that compared numbers are taken from the literature under their own protocols. That disclosure is exactly what a reviewer needs, and it is more than most papers in this area provide.

Critical weaknesses

W1. The headline ranking is not established by a controlled comparison. The paper states that "results of compared methods are taken from published literature under their own budgets, protocols, and full-model references" (§4.1). Because retention is normalized by each method's own full-model reference and the underlying harnesses, prompts and evaluation scripts differ, the 0.5–2.5 point gaps that separate CoViST from VisionTrim, MMTok, DeSAP and ZOO-Prune are of the same order as the harness differences themselves. Only one small reimplementation appears (Table 25, VisionZip). "Highest retention at every budget" is the paper's central empirical claim, and as submitted it rests on a cross-paper comparison; re-running the top three baselines under one harness is what would settle it. W2. The "exceeds vanilla" claims are single-run and, as stated, implausible. The text highlights MME 1901 vs 1867 at 192 tokens, MMB 64.3 vs 64.0 and POPE 86.2 vs 85.8 — i.e. removing two-thirds of the visual tokens improves perception. No seeds, confidence intervals or repeated evaluations are reported anywhere, while most claimed gaps are ≤0.5 points. A 34-point MME difference is far more likely to reflect evaluation sensitivity than a genuine gain from compression, and presenting it as evidence of the method's benefit weakens the credibility of the rest. Either support it with ≥3 seeds (or bootstrap CIs over the benchmark) or remove the claim. W3. No failure analysis and a narrow applicability envelope. The paper reports only aggregate retention; it does not say which tasks, question types or images degrade at 64 tokens (the natural suspects are OCR/TextVQA, counting and small objects), so a practitioner cannot tell where the 1.9% loss is concentrated. The method also requires white-box per-layer access to attention statistics and value projections; compatibility with fused kernels is shown only through SDPA on one A800, and the per-layer overhead of SCA at K = 64 is folded into TTFT rather than broken out. Finally, only image-only, single-image models are tested; multi-image and video, where progressive compression is most used, are not covered.

Questions for authors
  1. Can you re-run VisionTrim, MMTok and DeSAP (and VisionZip) under your own harness on LLaVA-1.5-7B at 192/128/64 tokens? How much of the 0.5–2.5 point gap survives?
  2. Please report at least three seeds or bootstrap confidence intervals for the "exceeds vanilla" claims (MME 1901 vs 1867, MMB 64.3 vs 64.0, POPE 86.2 vs 85.8). If they are not significant, will you remove the claim?
  3. Which tasks and question types lose most at 64 tokens? A per-category error analysis (OCR, counting, small objects) would make the aggregate retention interpretable.
  4. What is the per-layer overhead of SCA at K = 64, and how does the method interact with flash-attention/fused kernels in a standard serving stack?
  5. Does the method require white-box per-layer attention statistics and value projections, i.e. is it applicable to API-served models? What is the cost of computing the head gains per new model?
  6. Appendix E indicates that the copied TextVQA references follow the OCR-augmented convention. Please state this alignment in the main text and confirm it holds for all seven benchmarks and all copied baselines.
10 comments

(W1&Q1 part 1) Thanks for highlighting this concern. As requested, we have reproduced and re-evaluated VisionTrim, MMTok, DeSAP, VisionZip, ZOO-Prune, and RESTORE+HoloV under a unified harness on LLaVA-1.5-7B at 192/128/64 tokens. Specifically, we use the official released code for VisionTrim, MMTok, VisionZip, ZOO-Prune, and RESTORE. For DeSAP, we use our implementation based on its paper. We have tried our best to align the implementations with the papers. We also expand the evaluation from seven to eleven benchmarks by adding MMBench-CN (MMB-CN), SEEDBench-Image (SEED-I), MMMU, and VizWiz. The results are shown in the tables below, with both paper-reported and our-reproduced results presented. R7 and R11 denote the average performance retention over the original seven benchmarks and the expanded eleven benchmarks, respectively. For paper-reported results, retentions are calculated using each paper's corresponding uncompressed reference. Rank refers to R11 in this comparison. A dash indicates an unavailable result, and R11 and Rank are left blank for paper-reported rows with incomplete benchmark coverage.

Retain 192 tokens:

RankMethodGQAMMBMMEPOPESQATextVQAVQAv2MMB-CNSEED-IMMMUVizWizR7 (%)R11 (%)
—LLaVA-1.5-7B62.064.0186785.869.558.276.755.566.236.050.5100.0100.0
1CoViST-Pro61.664.5185586.269.357.976.554.565.935.749.499.899.4
2CoViST-Fixed61.364.3190186.269.157.676.454.865.435.649.099.999.3
3RESTORE+HoloV (reproduced)60.864.4179586.369.157.276.353.365.635.451.098.998.8
4ZOO-Prune (reproduced)60.062.9178687.369.057.175.452.964.335.852.598.398.6
5MMTok (reproduced)60.263.5178586.768.657.475.551.964.435.952.598.498.5
6DeSAP (reproduced)59.963.9177785.368.857.175.552.764.436.152.098.198.4
7VisionZip (reproduced)59.263.6176585.568.457.475.053.463.336.251.597.798.0
8VisionTrim (reproduced)58.363.1177783.868.757.374.254.562.436.151.597.197.7
—VisionTrim (paper)61.064.4179886.870.858.478.4———51.399.7—
—DeSAP (paper)60.363.5184887.169.957.277.5———54.599.1—
—RESTORE+HoloV (paper)61.063.7179386.669.657.277.6————98.8—
—ZOO-Prune (paper)60.062.9178287.269.257.377.3——36.1—98.3—
—MMTok (paper)60.163.4177486.468.857.777.1——36.3—98.2—
—VisionZip (paper)59.363.0178385.368.957.376.8——36.6—97.7—

Retain 128 tokens:

RankMethodGQAMMBMMEPOPESQATextVQAVQAv2MMB-CNSEED-IMMMUVizWizR7 (%)R11 (%)
—LLaVA-1.5-7B62.064.0186785.869.558.276.755.566.236.050.5100.0100.0
1CoViST-Pro61.364.7189086.169.157.776.354.065.535.449.199.999.2
2CoViST-Fixed61.364.1186786.369.057.176.253.365.334.949.099.598.6
3RESTORE+HoloV (reproduced)60.963.9178886.168.956.875.852.465.435.051.098.598.3
4ZOO-Prune (reproduced)59.661.6170887.069.256.974.653.763.035.853.197.197.9
5DeSAP (reproduced)59.163.3177983.668.956.774.652.163.236.153.097.397.8
6MMTok (reproduced)59.662.7175486.269.456.974.752.063.235.152.997.697.7
7VisionZip (reproduced)57.662.4176883.068.756.873.853.661.437.252.196.497.3
8VisionTrim (reproduced)56.461.6174979.968.656.672.453.659.836.951.994.996.1
—VisionTrim (paper)60.364.0178886.669.758.278.2———51.499.0—
—DeSAP (paper)59.562.8181587.069.957.076.9———54.398.3—
—RESTORE+HoloV (paper)60.863.0180786.068.756.677.1————98.2—
—ZOO-Prune (paper)59.561.9175287.168.957.976.6——35.7—97.6—
—MMTok (paper)59.362.3177986.368.857.076.4——35.7—97.5—
—VisionZip (paper)57.662.0176283.268.956.875.6——37.9—96.2—

(W1&Q1 part 2) Retain 64 tokens:

RankMethodGQAMMBMMEPOPESQATextVQAVQAv2MMB-CNSEED-IMMMUVizWizR7 (%)R11 (%)
—LLaVA-1.5-7B62.064.0186785.869.558.276.755.566.236.050.5100.0100.0
1CoViST-Pro60.964.4186386.068.457.075.653.264.735.249.199.198.4
2CoViST-Fixed60.263.1181985.868.556.475.052.264.135.348.898.197.5
3RESTORE+HoloV (reproduced)58.962.6177084.868.255.574.450.363.934.951.096.796.6
4MMTok (reproduced)58.261.6169286.069.155.873.549.061.535.253.896.096.2
5ZOO-Prune (reproduced)58.660.1164785.968.155.473.150.361.034.853.595.095.5
6DeSAP (reproduced)57.461.0170680.769.255.272.848.861.235.253.794.795.2
7VisionZip (reproduced)55.159.8169876.869.055.470.650.957.836.053.092.893.9
8VisionTrim (reproduced)53.459.5163270.468.655.268.051.555.135.952.590.191.9
—VisionTrim (paper)58.863.0178086.271.056.876.8———52.398.0—
—DeSAP (paper)58.362.3174885.869.655.776.7———53.796.8—
—RESTORE+HoloV (paper)59.061.9178784.968.055.475.6————96.5—
—MMTok (paper)58.361.2171585.869.256.075.2—61.336.1—96.0—
—ZOO-Prune (paper)58.560.2167685.968.355.475.0——35.4—95.2—
—VisionZip (paper)55.160.1169077.069.055.572.4——36.2—92.7—

From these results, it can be observed that the reproduced MMTok, VisionZip, ZOO-Prune, and RESTORE achieve similar performance with their papers. Our reproduced performance is sometimes even better (e.g. MMTok at 192/128, VisionZip at 128/64). For VisionTrim and DeSAP, the reproduced performance can be significantly lower. However, it seems that no researcher can reproduce VisionTrim using its official code according to the discussion from issues of their GitHub repository. Nevertheless, these results don't influence the performance retention ranking in our paper, as CoViST-Fixed/Pro still outperform the compared methods. In fact, our reproductions suggest that CoViST's performance advantage may be substantially larger than reported in our original paper. The following table compares the R7 performance gaps of CoViST-Fixed and CoViST-Pro over VisionTrim and DeSAP, and RESTORE+HoloV, three most competitive methods, using both their reported and reproduced results:

TokenBaselineCoViST-Fixed vs. PaperCoViST-Fixed vs. ReproducedCoViST-Pro vs. PaperCoViST-Pro vs. Reproduced
192VisionTrim+0.2+2.8+0.1+2.7
192DeSAP+0.8+1.8+0.7+1.7
192RESTORE+HoloV+1.1+1.0+1.0+0.9
128VisionTrim+0.5+4.6+0.9+5.0
128DeSAP+1.1+2.2+1.6+2.7
128RESTORE+HoloV+1.3+0.9+1.8+1.4
64VisionTrim+0.1+7.9+1.1+9.0
64DeSAP+1.2+3.4+2.3+4.5
64RESTORE+HoloV+1.6+1.4+2.7+2.4

According to these results, CoViST's performance advantage is even larger except for RESTORE. Considering that adopting the reproduced yet weaker results may overstate CoViST's advantage, we still prefer to adopt the reported results for compared methods from their papers. It should be emphasized that all compared methods are using their corresponding uncompressed performance under their evaluation settings as their reference, and therefore the comparison and ranking for performance retention are fair, as the differences caused by evaluation harness, hardware, implementation, and other factors, are largely mitigated.

We have added these additional results to the appendix E.2 of the revised paper.

(W2&Q2) Thanks for the valuable comment. We use lmms-eval to test our method, like many other works do. This library sets Python random seed to 0, NumPy seed to 1234, and PyTorch seed to 1234 by default, which we don't change, and thereby our reported results in the paper are using these seeds. We re-run our method using differents seeds, and the results are shown in the table below.

Python random seedNumPy seedPyTorch seedMMEMMBPOPEGQASQATextVQAVQAv2
012341234190164.386.261.369.157.676.4
424242189964.386.261.369.157.676.4
969696190064.386.161.269.157.576.4
199501291995012919950129190164.386.261.369.157.676.4
199605191996051919960519190064.386.261.369.157.676.4

It can be observed that the performance is nearly identical across all benchmarks, with minor differences largely confined to the second decimal place (e.g., 86.16 vs. 86.17 on POPE and 61.27 vs. 61.29 on GQA). Although the results under different seeds still exceed vanilla on MME, MMB, and POPE, our primary objective is not to surpass the vanilla baseline, but rather to maximize task performance under given visual token budgets. We don't intend to emphasize the "exceeds vanilla" claim, but just report and decribe the experimental results as observed. As mentioned in the conclusion, we plan to study this performance behavior more deeply in the future.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.