- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies visual token compression for text-rich tasks, including OCR and document understanding. It observes that existing training-free methods can underperform simple fixed pooling when the latter is coupled with model adaptation. The proposed VLVLM learns boundaries that divide the visual token sequence into variable-length groups and mean-pools each group before the language model. A lightweight MLP predicts boundaries, trained using Gumbel-Sigmoid relaxation and a compression-rate loss alongside language-model adaptation. At inference, top-K boundary selection enforces a specified token budget. Experiments cover multiple compression ratios, vision encoders, language backbones, and text-centric and general VQA benchmarks.
Critical strengths
-
A useful problem formulation and empirical observation. The paper highlights the sensitivity of OCR and document-understanding tasks to visual token compression. Showing that fixed pooling with adaptation can outperform the evaluated training-free methods is informative: it establishes the importance of adapting the language model to compressed visual inputs and provides a meaningful reference for evaluating more elaborate compression mechanisms.
-
Broad evaluation with a relevant adaptation baseline. The experiments include a trained fixed-pooling baseline, multiple compression ratios and backbones, and both text-centric and general tasks. Results at 8× compression, particularly with the Qwen2.5-14B backbone, provide evidence that learned grouping can improve over fixed pooling. These comparisons give the proposed design empirical support, even though the improvements vary across settings.
Critical weaknesses
-
The incremental methodological contribution is insufficiently established. Boundary prediction, variable-length mean pooling, Gumbel-Sigmoid training, and binomial rate regularization closely follow dynamic token pooling from Nawrot et al. (ACL 2023). The paper acknowledges this lineage and compares against an H-Net-style boundary predictor, but the main changes appear to be placement after the vision encoder and budget-controlled top-K inference. The empirical benefit of this adaptation is also uneven relative to fixed pooling trained under the same adaptation regime. In Table 1, OCR average retention improves from 86.80% to 88.65% at 4× and from 80.09% to 81.60% at 10×. With SigLIP2 at 4×, VLVLM underperforms fixed pooling (87.67% versus 88.47%); with Qwen2.5-14B at 10×, the improvement is only 0.37 retention percentage points. The stronger gains at 8× are valuable, but do not establish a consistently substantial benefit across the evaluated regimes. Consequently, I find the problem diagnosis and the importance of adaptation more convincing than the additional contribution of the proposed boundary-learning mechanism.
-
The practical efficiency advantage and competitiveness are not sufficiently demonstrated. The efficiency analysis reports forward-pass FLOPs, KV-cache size, and GPU memory, but lacks measured end-to-end inference latency, time to first token, or throughput. The FlashAttention timing experiment compares pretraining with and without FlashAttention; it does not establish VLVLM’s inference advantage over fixed pooling or other compression methods. Dynamic boundary selection and grouping may introduce overhead that FLOPs alone cannot characterize. In addition, the training-based comparisons mainly cover fixed pooling and Perceiver, leaving competitiveness against representative recent learned visual token compression methods unclear. These gaps limit assessment of the method’s practical value relative to its required adaptation and added mechanism.
Questions for authors
-
Contribution beyond existing dynamic pooling: Which design elements provide the substantive contribution beyond Nawrot et al.’s dynamic token pooling and related dynamic-chunking approaches? Beyond applying the mechanism after the vision encoder and using top-K inference, what existing evidence isolates a new capability or a distinct benefit specific to visual compression?
-
Robustness of the incremental gains: How do you explain the weaker or negative gains over trained fixed pooling at SigLIP2 4× and Qwen2.5-14B 10×, alongside the stronger gains at 8×? If available, please provide variation across training runs or other evidence supporting the robustness of this pattern.
-
Measured inference efficiency: Can you report end-to-end inference latency, time to first token, and throughput against fixed pooling and the evaluated compression baselines, including boundary prediction, token grouping, and generation? Please specify hardware, batch size, input resolution, and output length, and clarify the incremental training cost relative to fixed pooling.
-
Coverage of competitive baselines: Could you justify the limited coverage of recent learned visual token compression methods? Which representative methods are most directly comparable, and does existing evidence support the reported accuracy-efficiency advantage when model checkpoints, token budgets, training data, adaptation budgets, and evaluation conditions are aligned?
4 comments
for reviewer u9A8 (part 2: latency measurement; more baselines)
Measured inference efficiency: Can you report end-to-end inference latency, time to first token, and throughput against fixed pooling and the evaluated compression baselines, including boundary prediction, token grouping, and generation? Please specify hardware, batch size, input resolution, and output length, and clarify the incremental training cost relative to fixed pooling.
Please feel free to refer to our General Comments "Prefill latency improvement", where we measure the prefill latency gains on both LLaVA and Qwen3VL across multiple baselines and benchmarks. Under 4x compression, most methods and baselines including our VLVLM can reach up to 2x prefill latency speedup, for both LLaVA 1.5 7B and Qwen3VL 4B, where we also consider batch inference (batch size = 4) for Qwen3VL.
Coverage of competitive baselines: Could you justify the limited coverage of recent learned visual token compression methods? Which representative methods are most directly comparable, and does existing evidence support the reported accuracy-efficiency advantage when model checkpoints, token budgets, training data, adaptation budgets, and evaluation conditions are aligned?
You can feel free to refer to our General Comments "A broader coverage of training-free and training-based baselines". For training-free baselines, in addition to prompt-agnostic PruMerge and PruneSID, we also conduct experiments with one prompt-aware training-free baseline, SparseVLM. We find that our VLVLM consistently outperforms SparseVLM method, where we observe significant gains under aggressive compression budget (8x). For training-based baselines, other than our default 1D fixed pooling, we extend our experiments on spatially local 2D pooling. The results demonstrate that 2D fixed pooling performs similarly as 1D fixed pooling, and our VLVLM maintains the best OCR-heavy task performance. We also compare with another training-based baseline, TokenPacker. Please feel free to refer to our General Comments "A broader coverage of training-free and training-based baselines" for results.
Thank you again for your effort to review our work. We hope you could spend some time reviewing our rebuttal, and we are always happy to further clarify anything needed. If our rebuttal can address your concerns, we would appreciate it if you may reflect that in an updated score.
I thank the authors for the detailed rebuttal. The additional explanations adequately address my concerns regarding the motivation and novelty. However, I remain concerned about the empirical effectiveness. The improvements over trained fixed pooling are still small and inconsistent across settings, and most differences in the reported LLaVA-1.5 significance tests are not statistically significant. While adding SparseVLM is useful, SparseVLM was first released on arXiv in October 2024, over two years ago. This comparison alone therefore does not adequately cover more recent training-free methods or establish an advantage over strong training-free alternatives. The coverage and alignment of training-based comparisons also remain insufficient to isolate the benefit of variable-length grouping. Moreover, the reported OCRBench degradation in the Qwen3-VL-4B experiment (from 83.9 to 54.6) raises further concerns about practical applicability. I therefore maintain my original score.
Qwen3-VL-4B performance
We thank a lot for Reviewer u9A8 reply. In terms of reported OCRBench degradation and practical applicability, we clarify that the perceived "severe" degradation stems from evaluating on difficult, find-grained, text-centric benchmarks only. In addition, continued finetuning on Qwen3VL hurts the text-centric performance for all baselines, and hurts our VLVLM the least. In our General Comment "Coarse-grained benchmark results for Qwen3VL", we also evaluate our Qwen3VL experiment on coarse-grained general VQA tasks, covering benchmarks such as MMBench, MME, GQA, and POPE. We observe that on these simple benchmarks, all methods including VLVLM and representative baselines are performing well (almost lossless performance), revealing applicability on modern state-of-the-art VLMs. Besides, it is worth noting that our Qwen3VL experiment is a continued finetuning experiment, not training the model from scratch, which is the way our VLVLM is originally designed. We also believe that if we could train Qwen3VL from scratch with our method, it would lead to even less performance degradation.
for reviewer u9A8 (part 1: novelty and contribution; result explanation)
We thank reviewer
u9A8for feedback and comments. Here is how we address the raised weaknesses and concerns:We want to emphasize the fact that adopting the similar dynamic token pooling idea from NLP to vision is not trivial, where all of our design decisions are made specific to visual compression. Please feel free to refer to our General Comments "Novelty and contribution". We also highlight our contribution beyond algorithm and methodology, including benchmark performance, inference efficiency, and training data efficiency, elaborated in our General Comments.
For SigLIP2 4×, SigLIP2 model is trained with “local-to-global correspondence learning with self-distillation and masked prediction losses to improve the local semantics of the (un-pooled) feature representation” [1]. The improved un-pooled feature representation may make the model less sensitive to fixed pooling while maintaining good performance. Nevertheless, we show that under more aggressive compression (8× and 10×), our method VLVLM leads to much better performance when adaptive pooling matters more than the semantics. For Qwen2.5-14B 10×, since Qwen2.5-14B is a much strong language model, when the visual information is extremely compressed and limited (10×), in some tasks the VLM may rely more on the language prior from the LLM, leading to smaller gains. Nevertheless, our VLVLM still outperforms fixed pooling across 4 out of 5 text-centric benchmarks, as well as on coarse-grained benchmarks (94.57% v.s. fixed pooling 93.17% on average).
In terms of robustness of our results, for all experiments, we already apply McNemar's test to assess whether the observed performance improvements over the baselines are statistically significant to measure the robustness of the incremental gains over trained fixed pooling. We reveal the raw test statistics and corresponding p-values for LLaVA 1.5 experiment as a more detailed example here:
On benchmark DocVQA in particular, our method VLVLM's performance gain is consistently statistically significant.
References
[1] SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features.