acceptodds
Withdrawn from ICLR 2027

VLVLM: Learning Variable Length Visual Tokens for Efficient Vision Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Image tokens account for much of the compute cost of vision language models (VLMs), particularly at high resolution. Prior research has sought to reduce this cost by pruning tokens deemed redundant or uninformative, primarily through training-free approaches focusing on general visual understanding tasks. In this work, we show that these approaches incur substantial accuracy losses on text-rich images, where they are outperformed by a simple finetuning strategy that pools consecutive fixed-size groups of image tokens. However, fixed pooling is fundamentally limited in that it compresses all regions uniformly, despite differences in the amount of information they might contain. We introduce VLVLM, a training-based approach that dynamically chunks image tokens using a lightweight boundary predictor. At 4× compression (i.e., using only 25% of the original image tokens), VLVLM retains 89% of the uncompressed models’ performance on text-heavy visual benchmarks and outperforms both fixed pooling (87%) and training-free compression (82%), while preserving 99% performance on general tasks. Notably, the advantage persists under a more aggressive 8×compression: we obtain 85% retention compared with 81% for fixed pooling. These gains hold across VLM families, image encoders, and language model sizes (up to 14B). Further analyses show that the learned boundaries are semantically meaningful, aligning closely with text-rich regions. We will release all our code and checkpoints upon acceptance.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Tokenization, Efficiency, Mutimodal LLMs

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_u9A8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies visual token compression for text-rich tasks, including OCR and document understanding. It observes that existing training-free methods can underperform simple fixed pooling when the latter is coupled with model adaptation. The proposed VLVLM learns boundaries that divide the visual token sequence into variable-length groups and mean-pools each group before the language model. A lightweight MLP predicts boundaries, trained using Gumbel-Sigmoid relaxation and a compression-rate loss alongside language-model adaptation. At inference, top-K boundary selection enforces a specified token budget. Experiments cover multiple compression ratios, vision encoders, language backbones, and text-centric and general VQA benchmarks.

Critical strengths
  1. A useful problem formulation and empirical observation. The paper highlights the sensitivity of OCR and document-understanding tasks to visual token compression. Showing that fixed pooling with adaptation can outperform the evaluated training-free methods is informative: it establishes the importance of adapting the language model to compressed visual inputs and provides a meaningful reference for evaluating more elaborate compression mechanisms.

  2. Broad evaluation with a relevant adaptation baseline. The experiments include a trained fixed-pooling baseline, multiple compression ratios and backbones, and both text-centric and general tasks. Results at 8× compression, particularly with the Qwen2.5-14B backbone, provide evidence that learned grouping can improve over fixed pooling. These comparisons give the proposed design empirical support, even though the improvements vary across settings.

Critical weaknesses
  1. The incremental methodological contribution is insufficiently established. Boundary prediction, variable-length mean pooling, Gumbel-Sigmoid training, and binomial rate regularization closely follow dynamic token pooling from Nawrot et al. (ACL 2023). The paper acknowledges this lineage and compares against an H-Net-style boundary predictor, but the main changes appear to be placement after the vision encoder and budget-controlled top-K inference. The empirical benefit of this adaptation is also uneven relative to fixed pooling trained under the same adaptation regime. In Table 1, OCR average retention improves from 86.80% to 88.65% at 4× and from 80.09% to 81.60% at 10×. With SigLIP2 at 4×, VLVLM underperforms fixed pooling (87.67% versus 88.47%); with Qwen2.5-14B at 10×, the improvement is only 0.37 retention percentage points. The stronger gains at 8× are valuable, but do not establish a consistently substantial benefit across the evaluated regimes. Consequently, I find the problem diagnosis and the importance of adaptation more convincing than the additional contribution of the proposed boundary-learning mechanism.

  2. The practical efficiency advantage and competitiveness are not sufficiently demonstrated. The efficiency analysis reports forward-pass FLOPs, KV-cache size, and GPU memory, but lacks measured end-to-end inference latency, time to first token, or throughput. The FlashAttention timing experiment compares pretraining with and without FlashAttention; it does not establish VLVLM’s inference advantage over fixed pooling or other compression methods. Dynamic boundary selection and grouping may introduce overhead that FLOPs alone cannot characterize. In addition, the training-based comparisons mainly cover fixed pooling and Perceiver, leaving competitiveness against representative recent learned visual token compression methods unclear. These gaps limit assessment of the method’s practical value relative to its required adaptation and added mechanism.

Questions for authors
  1. Contribution beyond existing dynamic pooling: Which design elements provide the substantive contribution beyond Nawrot et al.’s dynamic token pooling and related dynamic-chunking approaches? Beyond applying the mechanism after the vision encoder and using top-K inference, what existing evidence isolates a new capability or a distinct benefit specific to visual compression?

  2. Robustness of the incremental gains: How do you explain the weaker or negative gains over trained fixed pooling at SigLIP2 4× and Qwen2.5-14B 10×, alongside the stronger gains at 8×? If available, please provide variation across training runs or other evidence supporting the robustness of this pattern.

  3. Measured inference efficiency: Can you report end-to-end inference latency, time to first token, and throughput against fixed pooling and the evaluated compression baselines, including boundary prediction, token grouping, and generation? Please specify hardware, batch size, input resolution, and output length, and clarify the incremental training cost relative to fixed pooling.

  4. Coverage of competitive baselines: Could you justify the limited coverage of recent learned visual token compression methods? Which representative methods are most directly comparable, and does existing evidence support the reported accuracy-efficiency advantage when model checkpoints, token budgets, training data, adaptation budgets, and evaluation conditions are aligned?

4 comments
AuthorsOpenReview ↗

for reviewer u9A8 (part 1: novelty and contribution; result explanation)

We thank reviewer u9A8 for feedback and comments. Here is how we address the raised weaknesses and concerns:

Contribution beyond existing dynamic pooling: Which design elements provide the substantive contribution beyond Nawrot et al.’s dynamic token pooling and related dynamic-chunking approaches? Beyond applying the mechanism after the vision encoder and using top-K inference, what existing evidence isolates a new capability or a distinct benefit specific to visual compression?

We want to emphasize the fact that adopting the similar dynamic token pooling idea from NLP to vision is not trivial, where all of our design decisions are made specific to visual compression. Please feel free to refer to our General Comments "Novelty and contribution". We also highlight our contribution beyond algorithm and methodology, including benchmark performance, inference efficiency, and training data efficiency, elaborated in our General Comments.

Robustness of the incremental gains: How do you explain the weaker or negative gains over trained fixed pooling at SigLIP2 4× and Qwen2.5-14B 10×, alongside the stronger gains at 8×? If available, please provide variation across training runs or other evidence supporting the robustness of this pattern.

For SigLIP2 4×, SigLIP2 model is trained with “local-to-global correspondence learning with self-distillation and masked prediction losses to improve the local semantics of the (un-pooled) feature representation” [1]. The improved un-pooled feature representation may make the model less sensitive to fixed pooling while maintaining good performance. Nevertheless, we show that under more aggressive compression (8× and 10×), our method VLVLM leads to much better performance when adaptive pooling matters more than the semantics. For Qwen2.5-14B 10×, since Qwen2.5-14B is a much strong language model, when the visual information is extremely compressed and limited (10×), in some tasks the VLM may rely more on the language prior from the LLM, leading to smaller gains. Nevertheless, our VLVLM still outperforms fixed pooling across 4 out of 5 text-centric benchmarks, as well as on coarse-grained benchmarks (94.57% v.s. fixed pooling 93.17% on average).

In terms of robustness of our results, for all experiments, we already apply McNemar's test to assess whether the observed performance improvements over the baselines are statistically significant to measure the robustness of the incremental gains over trained fixed pooling. We reveal the raw test statistics and corresponding p-values for LLaVA 1.5 experiment as a more detailed example here:

methodTextVQAOCRBenchOCRBenchv2DocVQAChartQAPro
LLaVA-1.5-7B58.1618.523.825.6112.04
4x compression
fixed pooling55.7415.720.918.7811.10
VLVLM (ours)55.8416.520.920.3910.91
McNemar's test0.07332.1270.23730.300.179
p-value0.7860.1450.6260.0000.672
8x compression
fixed pooling54.0514.319.516.0010.81
VLVLM (ours)55.0515.620.117.5910.94
McNemar's test1.4813.250.57532.3970.455
p-value0.2230.0710.4480.0000.499

On benchmark DocVQA in particular, our method VLVLM's performance gain is consistently statistically significant.

References

[1] SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features.

AuthorsOpenReview ↗

for reviewer u9A8 (part 2: latency measurement; more baselines)

Measured inference efficiency: Can you report end-to-end inference latency, time to first token, and throughput against fixed pooling and the evaluated compression baselines, including boundary prediction, token grouping, and generation? Please specify hardware, batch size, input resolution, and output length, and clarify the incremental training cost relative to fixed pooling.

Please feel free to refer to our General Comments "Prefill latency improvement", where we measure the prefill latency gains on both LLaVA and Qwen3VL across multiple baselines and benchmarks. Under 4x compression, most methods and baselines including our VLVLM can reach up to 2x prefill latency speedup, for both LLaVA 1.5 7B and Qwen3VL 4B, where we also consider batch inference (batch size = 4) for Qwen3VL.

Coverage of competitive baselines: Could you justify the limited coverage of recent learned visual token compression methods? Which representative methods are most directly comparable, and does existing evidence support the reported accuracy-efficiency advantage when model checkpoints, token budgets, training data, adaptation budgets, and evaluation conditions are aligned?

You can feel free to refer to our General Comments "A broader coverage of training-free and training-based baselines". For training-free baselines, in addition to prompt-agnostic PruMerge and PruneSID, we also conduct experiments with one prompt-aware training-free baseline, SparseVLM. We find that our VLVLM consistently outperforms SparseVLM method, where we observe significant gains under aggressive compression budget (8x). For training-based baselines, other than our default 1D fixed pooling, we extend our experiments on spatially local 2D pooling. The results demonstrate that 2D fixed pooling performs similarly as 1D fixed pooling, and our VLVLM maintains the best OCR-heavy task performance. We also compare with another training-based baseline, TokenPacker. Please feel free to refer to our General Comments "A broader coverage of training-free and training-based baselines" for results.

Thank you again for your effort to review our work. We hope you could spend some time reviewing our rebuttal, and we are always happy to further clarify anything needed. If our rebuttal can address your concerns, we would appreciate it if you may reflect that in an updated score.

I thank the authors for the detailed rebuttal. The additional explanations adequately address my concerns regarding the motivation and novelty. However, I remain concerned about the empirical effectiveness. The improvements over trained fixed pooling are still small and inconsistent across settings, and most differences in the reported LLaVA-1.5 significance tests are not statistically significant. While adding SparseVLM is useful, SparseVLM was first released on arXiv in October 2024, over two years ago. This comparison alone therefore does not adequately cover more recent training-free methods or establish an advantage over strong training-free alternatives. The coverage and alignment of training-based comparisons also remain insufficient to isolate the benefit of variable-length grouping. Moreover, the reported OCRBench degradation in the Qwen3-VL-4B experiment (from 83.9 to 54.6) raises further concerns about practical applicability. I therefore maintain my original score.

AuthorsOpenReview ↗

Qwen3-VL-4B performance

We thank a lot for Reviewer u9A8 reply. In terms of reported OCRBench degradation and practical applicability, we clarify that the perceived "severe" degradation stems from evaluating on difficult, find-grained, text-centric benchmarks only. In addition, continued finetuning on Qwen3VL hurts the text-centric performance for all baselines, and hurts our VLVLM the least. In our General Comment "Coarse-grained benchmark results for Qwen3VL", we also evaluate our Qwen3VL experiment on coarse-grained general VQA tasks, covering benchmarks such as MMBench, MME, GQA, and POPE. We observe that on these simple benchmarks, all methods including VLVLM and representative baselines are performing well (almost lossless performance), revealing applicability on modern state-of-the-art VLMs. Besides, it is worth noting that our Qwen3VL experiment is a continued finetuning experiment, not training the model from scratch, which is the way our VLVLM is originally designed. We also believe that if we could train Qwen3VL from scratch with our method, it would lead to even less performance degradation.

Reviewer_FQoKOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces VLVLM, a training-based visual-token compression method that groups consecutive image tokens into variable-length chunks using a lightweight learned boundary predictor. Tokens within each chunk are average-pooled to reduce the number of visual tokens passed to the language model. The method is trained using a language modeling objective and an auxiliary loss to regulate the compression ratio. Experiments across different vision encoders and language models show modest improvements on several OCR-intensive benchmarks, particularly at higher compression ratios (8×–10×), although the gains are not consistent across all settings.

Critical strengths

None

Critical weaknesses
  1. Incomplete comparisons with training-based methods. The evaluation primarily compares against training-free methods, while the training-based baselines are limited to Perceiver and fixed pooling. Several relevant training-based visual-token compression methods, such as TokenPacker[1], VisionSelector[2], MetaCompress[3], and EvoComp[4], are omitted, making it difficult to assess the effectiveness of VLVLM against existing alternatives.
  2. Insufficient ablations to isolate the benefits of variable-length grouping. The fixed-pooling baseline operates on flattened visual tokens, potentially overlooking spatial locality. At the minimum, comparisons against trained 2D spatial pooling would help establish whether the improvements specifically arise from adaptive grouping.
  3. Weak benchmark performance and severe degradation on modern VLMs. The evaluation relies heavily on LLaVA-1.5-7B, which achieves only 15–30% on benchmarks such as OCRBench and DocVQA, substantially below the 80–90% performance achieved by modern VLMs. Improvements of just 1–2 points in such a low-performing regime raise questions about their practical significance. More concerningly, adapting Qwen3-VL-4B (Table 4) results in severe performance degradation, with OCRBench dropping from 83.9 to 54.6 despite fine-tuning. These results raise serious concerns about the applicability of the proposed method to modern VLMs.

[1] - TokenPacker: Efficient Visual Projector for Multimodal LLM [2] - VisionSelector: End-to-End Learnable Visual Token Compression for Efficient Multimodal LLMs [3] - Rethinking Token Reduction for Large Vision-Language Models [4] - EvoComp: Learning Visual Token Compression for Multimodal Large Language Models via Semantic-Guided Evolutionary Labeling

Questions for authors
  1. Can the authors compare VLVLM against established training-based compression methods such as TokenPacker, VisionSelector, MetaCompress, or EvoComp under matched training and compression budgets? Additionally, why was the fine-tuned variant of PruMerge not considered?
  2. Can the authors compare against trained 2D spatial pooling and learned Top-K token selection? These comparisons would help isolate whether the reported improvements arise specifically from variable-length grouping rather than from finetuning or learned token selection.
  3. Table 4 shows that applying VLVLM to Qwen3-VL-4B reduces OCRBench performance from 83.9 to 54.6 despite finetuning. What explains this substantial degradation, and how do the authors justify the practical utility of this compression approach when applied to modern VLMs?
  4. Qwen3-VL already employs a learned 2×2 spatial token merger and supports native-resolution inputs. Can the authors report actual time-to-first-token (TTFT) or end-to-end prefill latency on OCRBench (or any other benchmark), comparing the original Qwen3-VL with VLVLM under identical conditions? It would be useful to report the image resolutions, visual-token counts, and latency breakdown between vision encoding and LLM prefill to establish the practical benefits of additional compression beyond Qwen3-VL's existing token reduction.
2 comments
AuthorsOpenReview ↗

for reviewer FQoK (part 1: more baselines)

We thank for reviewer FQoK feedback and comments. Here is how we address the raised weaknesses and concerns:

Incomplete comparisons with training-based methods. The evaluation primarily compares against training-free methods, while the training-based baselines are limited to Perceiver and fixed pooling. Several relevant training-based visual-token compression methods, such as TokenPacker[1], VisionSelector[2], MetaCompress[3], and EvoComp[4], are omitted, making it difficult to assess the effectiveness of VLVLM against existing alternatives.

Most mentioned training-based baselines have their own limitations. LearnPruner, MetaCompress, and EvoComp do not have open-sourced code implementation or artifacts, meaning that their results cannot be reproduced. Nevertheless, we compare VLVLM with the training-based baseline, TokenPacker, whose code is available. Please feel free to refer to our General Comments "A broader coverage of training-free and training-based baselines" for results.

Insufficient ablations to isolate the benefits of variable-length grouping. The fixed-pooling baseline operates on flattened visual tokens, potentially overlooking spatial locality. At the minimum, comparisons against trained 2D spatial pooling would help establish whether the improvements specifically arise from adaptive grouping.

For spatially local 2D pooling controls, we also conduct experiments on training-based spatially local 2D pooling. Please feel free to refer to our General Comments "A broader coverage of training-free and training-based baselines", where our results demonstrate that 2D fixed pooling performs similarly as 1D fixed pooling, and our VLVLM maintains the best OCR-heavy task performance.

AuthorsOpenReview ↗

for reviewer FQoK (part 2: more Qwen3VL results; latency measurement)

Weak benchmark performance and severe degradation on modern VLMs. The evaluation relies heavily on LLaVA-1.5-7B, which achieves only 15–30% on benchmarks such as OCRBench and DocVQA, substantially below the 80–90% performance achieved by modern VLMs. Improvements of just 1–2 points in such a low-performing regime raise questions about their practical significance. More concerningly, adapting Qwen3-VL-4B (Table 4) results in severe performance degradation, with OCRBench dropping from 83.9 to 54.6 despite fine-tuning. These results raise serious concerns about the applicability of the proposed method to modern VLMs.

In terms of "weak benchmark performance", we also evaluate our method VLVLM against baselines on a more recent, real-world, high-resolution benchmark called MME-RealWorld, and you can feel free to refer to our General Comments "Evaluation on high-resolution benchmarks". We observe that not only does VLVLM outperform other baselines especially on task subsets like "OCR" and "Diagram and Table", more importantly, our method VLVLM (and our proposed fixed pooling baseline) actually exceed the performance of original, uncompressed LLaVA, highlighting its favorable practical use cases.

As for "severe degradation on modern VLMs", we clarify that the perceived "severe" degradation stems from evaluating on difficult, find-grained, text-centric benchmarks only. In addition, continued finetuning on Qwen3VL hurts the text-centric performance for all baselines, and hurts our VLVLM the least. In our General Comment "Coarse-grained benchmark results for Qwen3VL", we also evaluate our Qwen3VL experiment on coarse-grained general VQA tasks, covering benchmarks such as MMBench, MME, GQA, and POPE. We observe that on these simple benchmarks, all methods including VLVLM and representative baselines are performing well (almost lossless performance), revealing applicability on modern state-of-the-art VLMs.

Besides, it is worth noting that our Qwen3VL experiment is a continued finetuning experiment, not training the model from scratch, which is the way our VLVLM is originally designed. We also believe that if we could train Qwen3VL from scratch with our method, it would lead to even less performance degradation.

Qwen3-VL already employs a learned 2×2 spatial token merger and supports native-resolution inputs. Can the authors report actual time-to-first-token (TTFT) or end-to-end prefill latency on OCRBench (or any other benchmark), comparing the original Qwen3-VL with VLVLM under identical conditions? It would be useful to report the image resolutions, visual-token counts, and latency breakdown between vision encoding and LLM prefill to establish the practical benefits of additional compression beyond Qwen3-VL's existing token reduction.

Please feel free to refer to our General Comments "Prefill latency improvement", where we measure the prefill latency gains on both LLaVA and Qwen3VL across multiple baselines and benchmarks. Under 4x compression, most methods and baselines including our VLVLM can reach up to 2x prefill latency speedup for Qwen3VL 4B model.

Thank you again for your effort to review our work. We hope you could spend some time reviewing our rebuttal, and we are always happy to further clarify anything needed. If our rebuttal can address your concerns, we would appreciate it if you may reflect that in an updated score.

Reviewer_yzVqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

VLVLM inserts a boundary predictor between the vision encoder and projector, grouping consecutive visual tokens into variable-length chunks for average pooling. Task loss and a boundary-count objective train the model jointly, while top-k boundaries enforce inference budgets. The experiments highlight the importance of training adaptation and show further benefits from learned grouping, particularly on text-rich tasks at stronger compression.

Critical strengths

I am glad to see the paper first ask how much training adaptation matters, before asking whether boundaries need to be learned. Figure 2’s trained-versus-untrained fixed-pooling comparison, and the trained fixed-pooling baselines in Tables 1–3, avoid attributing every gain to the new module. This simple baseline is informative in its own right: a compression rule should not be judged only through a model that has not adapted to its outputs.

Critical weaknesses

I have presented the weaknesses together with the relevant details to make it easier for the authors to address them during the rebuttal period. If my concerns are adequately addressed, I would be willing to raise my score.

  1. My first concern is the treatment of the final chunk in Algorithms 2–3 of Appendix B. Algorithm 2 takes the K highest-probability boundaries without requiring the last token to be a boundary. Algorithm 3 allocates segments from the boundary count and retains only indices 0 through S−1. For a one-image batch with boundaries [0,1,0,0], the shifted cumulative indices are [0,0,1,1], but S=1, leaving the final two tokens outside every output segment. The implementation may force a terminal boundary or handle the remainder separately, but this is not stated. Please provide implementation-faithful pseudocode that guarantees both full coverage and the exact budget, and specify how soft and hard boundaries connect to the pooling loss through the straight-through estimator.

  2. In my view, Table 5’s text coverage indicates where chunk endpoints fall, not directly how much text information survives pooling. VLVLM averages non-boundary tokens too; an endpoint on text does not guarantee that the corresponding chunk avoids substantial background mixing. The paper itself finds that PruneSID sometimes has higher coverage but worse task performance. I would like chunk-length and text/background-mixing statistics for text-containing chunks, plus trained random variable-length boundaries or spatially local 2D pooling controls. Appendix E already acknowledges vertical-text failures; checking row boundaries and scan order would make the semantic-boundary interpretation more convincing.

  3. The fixed-pooling control is valuable, but PruMerge and PruneSID are still evaluated only as training-free methods. Since training–inference mismatch is central to the motivation, adapting those selectors with the same data and update budget is an especially relevant control. Perceiver does not fully resolve this issue because it changes the compression mechanism as well. LearnPruner is discussed but not directly compared. I would like at least one nonuniform selection or merging baseline with matched fine-tuning, to establish whether the advantage comes from variable-length consecutive grouping rather than primarily from adapting the language model to compressed inputs.

  4. The FLOPs, KV-cache, and memory results in Appendix C are useful, but Table 8’s 55-to-30-hour comparison mainly measures training with versus without FlashAttention, not inference acceleration over baselines using the same backend. Algorithm 3 also constructs a token-to-segment assignment matrix, and it is unclear whether the implementation uses a more efficient segmented reduction. Please report time to first token, end-to-end latency at fixed generation lengths, and the predictor/pooling overhead under matched settings, particularly for high-resolution Qwen3VL. This would establish whether the arithmetic savings translate into deployment benefits.

Minor Weaknesses:

  1. Reporting McNemar tests is welcome, but please define the paired binary correctness decision for each task, especially where evaluation can award partial credit. Repeated training seeds or paired confidence intervals would also help, since several 4× differences are small and do not consistently favor VLVLM.

  2. For 576 input tokens, Tables 1 and 3 list 58 retained tokens at 10×, whereas Table 2 lists 57 and Algorithm 2 uses integer(L·ρ). Please make the rounding convention consistent and confirm that baselines use the same budget. Figure 6 should also specify how its aggregate combines OCR and other tasks.

Questions for authors

For clarity and to facilitate the rebuttal process, I have already outlined the key strengths and weaknesses that informed my rating in the sections above. Please refer to the Strengths and Weaknesses sections for the detailed rationale. I would be willing to reconsider my rating if the authors adequately address my main concerns during the rebuttal.

2 comments
AuthorsOpenReview ↗

for reviewer yzVq (part 1: implementation details)

We thank reviewer yzVq feedback and comments. Here is how we address the raised weaknesses and concerns:

My first concern is the treatment of the final chunk in Algorithms 2–3 of Appendix B. Algorithm 2 takes the K highest-probability boundaries without requiring the last token to be a boundary. Algorithm 3 allocates segments from the boundary count and retains only indices 0 through S−1. For a one-image batch with boundaries [0,1,0,0], the shifted cumulative indices are [0,0,1,1], but S=1, leaving the final two tokens outside every output segment. The implementation may force a terminal boundary or handle the remainder separately, but this is not stated. Please provide implementation-faithful pseudocode that guarantees both full coverage and the exact budget, and specify how soft and hard boundaries connect to the pooling loss through the straight-through estimator.

For the terminal boundary, in order to match the exact budget, we actually allow the last segment to be dropped if the very last image token is not being selected as a boundary and it empirically affects little performance. Here is the exact code we use to match the inference budget:

# top-k routing: select exactly k boundaries per sample
_, topk_idx = torch.topk(soft_boundaries, k=k, dim=-1)
hard_boundaries = torch.zeros_like(soft_boundaries)
hard_boundaries.scatter_(dim=-1, index=topk_idx, value=1.0)

Besides, in our code implementation, we emphasize that the straight-through estimator is only used during training, where we obtain the hard boundaries via the standard straight-through estimator in order to train VLVLM with non-differentiable operations (i.e., boundary selection):

hard_boundaries = (soft_boundaries > 0.5).float()
hard_boundaries = hard_boundaries - soft_boundaries.detach() + soft_boundaries

In my view, Table 5’s text coverage indicates where chunk endpoints fall, not directly how much text information survives pooling. VLVLM averages non-boundary tokens too; an endpoint on text does not guarantee that the corresponding chunk avoids substantial background mixing. The paper itself finds that PruneSID sometimes has higher coverage but worse task performance. I would like chunk-length and text/background-mixing statistics for text-containing chunks, plus trained random variable-length boundaries or spatially local 2D pooling controls. Appendix E already acknowledges vertical-text failures; checking row boundaries and scan order would make the semantic-boundary interpretation more convincing.

Although our proposed VLVLM leads to background mixing due to the nature of token pooling, many more boundary tokens still end up being predicted around text-heavy regions, preserving its strong OCR-related task performance. Additionally, mixing the background is not necessarily a weakness: depending on the question prompt, it will be beneficial not to discard the background completely.

For spatially local 2D pooling controls, we also conduct experiments on training-based spatially local 2D pooling. Please feel free to refer to our General Comments "A broader coverage of training-free and training-based baselines", where our results demonstrate that 2D fixed pooling performs similarly as 1D fixed pooling, and our VLVLM maintains the best OCR-heavy task performance.

AuthorsOpenReview ↗

for reviewer yzVq (part 2: more baselines; latency measurement)

The fixed-pooling control is valuable, but PruMerge and PruneSID are still evaluated only as training-free methods. Since training–inference mismatch is central to the motivation, adapting those selectors with the same data and update budget is an especially relevant control. Perceiver does not fully resolve this issue because it changes the compression mechanism as well. LearnPruner is discussed but not directly compared. I would like at least one nonuniform selection or merging baseline with matched fine-tuning, to establish whether the advantage comes from variable-length consecutive grouping rather than primarily from adapting the language model to compressed inputs.

For training-based baselines, other than our default 1D fixed pooling, we extend our experiments on spatially local 2D pooling. We also compare with another training-based baseline, TokenPacker. Please feel free to refer to our General Comments "A broader coverage of training-free and training-based baselines" for results. LearnPruner is discussed but not directly compared because their code is not publicly available and its results cannot be reproduced. PruMerge and PruneSID are proposed as mainly training-free methods: PruMerge finetuning only leads to marginal gains according to what the paper reports (only 0.1% gain on POPE benchmark), and PruneSID does not even support batching at all, limiting its practical training use cases.

The FLOPs, KV-cache, and memory results in Appendix C are useful, but Table 8’s 55-to-30-hour comparison mainly measures training with versus without FlashAttention, not inference acceleration over baselines using the same backend. Algorithm 3 also constructs a token-to-segment assignment matrix, and it is unclear whether the implementation uses a more efficient segmented reduction. Please report time to first token, end-to-end latency at fixed generation lengths, and the predictor/pooling overhead under matched settings, particularly for high-resolution Qwen3VL. This would establish whether the arithmetic savings translate into deployment benefits.

Please feel free to refer to our General Comments "Prefill latency improvement", where we measure the prefill latency gains on both LLaVA and Qwen3VL across multiple baselines and benchmarks. Under 4x compression, most methods and baselines including our VLVLM can reach up to 2x prefill latency speedup, for both LLaVA 1.5 7B and Qwen3VL 4B.

Thank you again for your effort to review our work. We hope you could spend some time reviewing our rebuttal, and we are always happy to further clarify anything needed. If our rebuttal can address your concerns, we would appreciate it if you may reflect that in an updated score.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General Comments (part 1: Novelty and Contribution; Evaluation on high-resolution benchmarks)

Novelty and contribution

We want to emphasize the fact that adopting the similar dynamic token pooling idea from NLP (e.g., byte-level LMs) to vision is not trivial: in fact, all of our design decisions are made specific to visual compression. We elaborate on them as follows.

  1. Putting the boundary predictor after the vision encoder instead of within the vision encoder: (1) most of the vision encoders are already pretrained and kept frozen during multimodal instruction tuning. Importantly, our early experiments show that inserting a boundary predictor within the vision encoder severely degrades the performance (e.g., almost 30% drop on MMBench); (2) in modern state-of-the-art VLM architectures such as Qwen3VL, they use DeepStack [1] to stack in image features from shallow layers: a boundary predictor in between layers would lead to shape mismatch; (3) The main compute bottleneck in terms of both FLOPs and latency actually stem from LLM layers; saving computation within the vision encoder would only lead to very limited additional efficiency gains.

  2. Using top-K selection during inference: (1) the original dynamic token pooling work in NLP does not use top-K during inference, where instead it uses the same thresholding approach during its training. That is because the predicted boundary probabilities are quite confident (either close 1s or 0s) and applying thresholding leads to a similar compute budget. However, in our early experiments with LLaVA, we find that the predicted boundary probabilities are typically not confident enough; therefore, only top-K selection can guarantee the matched budget when comparing with baselines. We speculate that this may be lie in the attention pattern difference between causal-attention LMs and bidirectional-attention vision encoders. (2) using top-K selection can also allow for any arbitrary inference budget, as we have demonstrated in our paper's discussion section "you only train it once". This is very valuable since one can just train one VLVLM once, and set any inference budget on the fly. This particular benefit of achieving arbitrary inference budget is never enabled in the dynamic token pooling work and it's only possible when we use top-K selection during inference.

We also want to highlight our contribution beyond algorithm and methodology. (1) performance-wise, our proposed method is effective to improve VLM's capabilities on text-centric and fine-grained benchmarks, spanning different vision encoders, underlying LLMs, and VLM architectures; (2) inference efficiency-wise, our VLVLM achieves similar FLOP and latency gains as state-of-the-art baselines and the boundary predictor only introduces, as Table 7 in the appendix shows, 4.2M learned parameters on top of fixed pooling; (3) data efficiency-wise, as our discussion section "Can we Train with Fewer Samples?" illustrates, VLVLM achieves the same level of performance retention while using only 25% of the data required by fixed pooling (VLVLM @ 166.2K v.s. fixed 665K). Note that all of these contributions are also distinct benefits specific to visual compression.

Evaluation on high-resolution benchmarks

We also evaluate our method VLVLM against baselines on a more recent, real-world, high-resolution benchmark MME-RealWorld [2]:

ModelmonitoringOCRDiagram and TableAutonomous DrivingRemote SensingOverallAvg. Ret.
LLaVA19.2243.6220.2126.8922.7428.09100%
4x compression
PruMerge19.3142.6520.2726.9422.6327.8799.22%
PruneSID19.0343.1420.6126.6428.0928.12100.11%
fixed pooling21.9544.1620.4326.2024.4828.69102.14%
VLVLM (ours)20.2648.2421.7927.4926.8630.77109.54%
8x compression
PruMerge19.3142.4620.2726.9722.6627.8299.04%
PruneSID19.3142.8920.4926.6422.7428.0399.79%
fixed pooling20.9547.0620.8927.3526.5730.20107.51%
VLVLM (ours)21.8646.9521.4427.4326.9730.44108.37%

VLVLM outperforms other baselines such as PruMerge, PruneSID, and fixed pooling, under both moderate (4x) and aggressive (8x) compression budgets, especially on tasks subsets like "OCR" and "Diagram and Table". More importantly, both our method VLVLM and fixed pooling baseline actually exceed the performance of original, uncompressed LLaVA, highlighting favorable practical use cases.

References

[1] DeepStack: Deeply Stacking Visual Tokens is Surprisingly Simple and Effective for LMMs, NeurIPS 2024.

[2] MME-RealWorld: Could Your Multimodal LLM Challenge High-Resolution Real-World Scenarios that are Difficult for Humans?, ICLR 2025.

AuthorsOpenReview ↗

General Comments (part 2: A broader coverage of training-free and training-based baselines)

A broader coverage of training-free and training-based baselines

For training-free baselines, as suggested by u9A8, in addition to prompt-agnostic PruMerge and PruneSID, we also conduct experiments with one prompt-aware training-free baseline, SparseVLM. Here are the results on LLaVA 1.5:

methodTextVQAOCRBenchOCRBenchv2DocVQAChartQAProAvg. Ret. (OCR)
LLaVA-1.5-7B58.1618.523.825.6112.04100%
4x compression
SparseVLM55.1015.221.317.3311.4685.85%
VLVLM (ours)55.8416.520.920.3910.9188.65%
8x compression
SparseVLM52.9311.018.211.6210.9872.70%
VLVLM (ours)55.0515.620.117.5910.9484.59%

Note that our VLVLM consistently outperforms SparseVLM method, where we observe significant gains under aggressive compression budget (8x).

For training-based baselines, other than our default 1D fixed pooling, we extend our experiments on spatially local 2D pooling, as suggested by reviewer FQoK and yzVq. Here are the results on LLaVA 1.5:

methodTextVQAOCRBenchOCRBenchv2DocVQAChartQAProAvg. Ret. (OCR)
LLaVA-1.5-7B58.1618.523.825.6112.04100%
4x compression
fixed pooling55.7415.720.918.7811.1086.80%
fixed (2D)53.3715.721.918.6010.6785.97%
VLVLM (ours)55.8416.520.920.3910.9188.65%
8x compression
fixed pooling54.0514.319.516.0010.8180.88%
fixed (2D)51.5413.620.416.309.3977.89%
VLVLM (ours)55.0515.620.117.5910.9484.59%

The results demonstrate that 2D fixed pooling performs similarly as 1D fixed pooling, and our VLVLM maintains the best OCR-heavy task performance.

We also compare our VLVLM with another training-based baseline, TokenPacker, as suggested by reviewer FQoK. Note that unlike our VLVLM, TokenPacker is inflexible: it does not support any arbitrary compression ratio because it operates on a square-sized fixed window with cross attention on image features from intermediate vision encoder layers. Therefore, we compare under both 4x (i.e., 2x2) and 9x (i.e., 3x3) compression. For 9x compression, we use the 8x trained checkpoint for VLVLM to evaluate on 9x to match the same inference budget while avoiding retraining. We also include MME-Realworld in our evaluation below.

ModelTextVQAOCRBenchOCRBenchv2DocVQAChartQAProMME-RealWorld (overall)Avg. Ret.TFLOPs
LLaVA-1.5-7B58.1618.523.825.6112.0428.09100%8.89
4x compression
TokenPacker55.2316.822.119.779.0226.3787.44%3.18
VLVLM (ours)55.8416.520.920.3910.9130.7792.13%3.18
9x compression
TokenPacker53.2515.120.616.789.6326.42 83.22%2.23
VLVLM (ours)54.7715.120.417.4210.9030.4088.05%2.23

A detailed subset performance breakdown of MME-Realworld:

ModelmonitoringOCRDiagram and TableAutonomous DrivingRemote SensingOverallAvg. Ret.
LLaVA19.2243.6220.2126.8922.7428.09100%
4x compression
TokenPacker19.08 38.14 20.08  26.04  22.04 26.3793.88%
VLVLM (ours)20.2648.2421.7927.4926.8630.77109.54%
9x compression
TokenPacker19.08 38.31 20.08 26.04 22.04 26.42 94.05% 
VLVLM (ours)21.8646.7621.4227.0526.6230.40108.22%

We observe that VLVLM outperforms TokenPacker on average, especially on the high-resolution benchmark MME-realworld. Other training-based baselines mentioned by FQoK are not open-sourced and results cannot be faithfully reproduced.

AuthorsOpenReview ↗

General Comments (part 3: Prefill latency improvement; Coarse-grained benchmark results for Qwen3VL)

Prefill latency improvement

We measure the LLM prefill latency (ms) to demonstrate the efficiency gain beyond FLOPs, spanning a diverse collection of benchmarks (TextVQA, DocVQA for OCR-heavy tasks, MME, GQA, POPE, AND MM-Vet or general VQA tasks):

MethodMMETextVQADocVQAGQAPOPEMM-Vet
LLaVA-1.5 7B60.0265.7957.3759.3155.5955.72
4x compression
PruMerge29.8232.1229.1729.3729.4229.83
PruneSID28.9632.3729.0729.1929.2028.99
SparseVLM42.4446.3441.0341.6542.7541.81
Fixed pooling29.8133.3429.9330.0330.0329.73
VLVLM29.6233.3430.0030.2329.7829.82

Under 4x compression, most methods and baselines including VLVLM can reach up to 2x prefill latency speedup with a single batch size; SparseVLM only reaches up to 1.5x latency boost since it starts merging image tokens during the LLM layer.

To validate the latency gains with more modern VLMs, we further measure the prefill latency improvement on multiple benchmarks including OCRBench, TextVQA, and ChartQAPro for Qwen3VL 4B model. We first conduct a profiling experiment on a single-sample (batch size = 1) setting, and VLVLM achieves similar prefill speedup as other baselines such as fixed pooling and PruneSID. To maximize GPU utilization, we also consider a batched (batch size = 4) setting for inference, which is not supported by PruneSID.

MethodOCRBenchTextVQAChartQAProOCRBenchTextVQAChartQAPro
batch size = 1batch size = 4
Qwen3VL-4B92.0269.50131.79406.78299.09830.45
4x compression
PruneSID57.7753.5463.62------
Fixed pooling57.7653.8763.49123.85147.11216.54
VLVLM58.4554.0063.02124.36143.52215.41

Coarse-grained benchmark results for Qwen3VL

We evaluate our Qwen3VL experiment on coarse-grained general VQA tasks, covering benchmarks such as MMBench, MME, GQA, and POPE. We observe that on these simple benchmarks, all methods including VLVLM and representative baselines are performing well, revealing applicability on modern state-of-the-art VLMs.

MethodMMBenchMMEGQAPOPE
Qwen3VL-4B83.901710.4862.4989.66
SFT@66K88.421729.5462.9490.22
Additional 4x compression
PruneSID85.631638.4161.1389.35
Fixed pooling@66K85.311668.4460.4188.90
VLVLM@66K84.971680.8959.5489.22

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.