The work has the following primary weaknesses:
- W1: Novelty and Missing Comparisons
- As it stands, there are several relevant works that the work is not discussing or comparing against, but should have. To exemplify, the authors only cite but do not discuss or compare against Q-DOVE [A], which is an especially relevant work as it also has a similar text-conditioned visual tokenization strategy with a very similar reconstruction strategy where it weights reconstruction toward relevant image regions and producing a variable number of tokens. At the very least, the authors should discuss in detail how their work differs from it, and ideally provide qualitative and quantitative comparisons against it.
- The work also cites MQT [B], which is another query-based elastic compression method but has no comparisons against it, while failing to cite another similar elastic compression competitor based on pooling altogether, M$^3$ [C]. Given both baselines are training-based elastic compression methods, the authors should ideally empirically compare QART against them.
- W2: Problems with Experiments, Evaluation Fairness and Breadth
- For vision-language understanding under compression, the work primarily relies on two benchmarks: VQAv2 and Visual-7W, and there are several issues around both the evaluation fairness and breadth across these settings.
- For VQAv2, many baselines reported in Table 1 are training-free and/or were initially proposed for CLIP-based LLaVA 1.5, whereas QART requires additional training with dense annotations while utilizing SigLIP-2. This renders the existing comparisons not fully-apples-to-apples. Although I appreciate the inclusion of one such SigLIP-2 comparison in the table, the authors do not report a QART variant with a CLIP-based LLaVA 1.5, matching the original set-up of the baselines. Accordingly, at least including such a setting, clearly marking which baselines are training-free, and including more training-based baselines (e.g. those mentioned under W1), would be a better way to benchmark QART.
- For Visual7W, given both (i) most, if not all, of these benchmarks omit this benchmark and (ii) all are adapted with a SigLIP-2 backbone, the authors should describe precisely and in detail how were the baseline settings transferred. The Appendix of the work states that all learnable baselines were given an additional 4 more epochs of training, though it is unclear how this training duration was determined and whether a comparable tuning to that of QART was performed for each of the baselines. In any future revisions, this aspect should ideally be clarified in detail.
- More importantly than the above items, the evaluation presented in the work is very limited for a method aiming to improve the visual tokenization strategy of generalist LVLMs. From the two benchmarks, VQAv2 is known to be a relatively easy benchmark to retain strong performance with low-resolution inputs or smaller number of visual tokens [D, E, F, G] and its training images are included in the standard LLaVA training set-up [B, C, H], making benchmarking there show no indication of zero-shot performance. Furthermore, Visual7W falls short of fully addressing this issue (the precise issue with it is described under the following item). Accordingly, the authors should report results with significantly more and diverse benchmarks, including but not limited to TextVQA, DocVQA, ChartQA, TextCaps and more.
- Visual7W and RefCOCO are based on the images of the same dataset, MS COCO. The authors should thus clarify precisely whether the Visual7W evaluation images they are reporting the results on were initially present in the training data. On a similar note, the use of RefCOCO during training also blocks the authors from reporting results on that particular benchmark, which is a well-used benchmark for benchmarking LVLMs under visual token compression [F]. Ideally, to test the grounding capabilities of the proposed approach, some comparisons on RefCOCO should be reported and compared against baselines, e.g. against FEATHER [F], after removing RefCOCO data from the training mixture of QART.
- Finally, both the LVLMs utilized by the authors are based on significantly older LLMs. Ideally, at least some newer backbone should be used for reporting results and under more diverse evaluation settings, such as a member of the newer Qwen/InternVL model families.
- W3: Concerns with Added Computational Costs
- Comparing QART and the baselines primarily based on the number of visual tokens is unfair, as it is not established whether they produce the same or similar computational costs, e.g. w.r.t. inference FLOPs, KV cache usage or latency. Although I acknowledge that Figure 4 reports E2E inference results for QART and the base model, the relevance predictor and the additional processing have at least some costs beyond the number of visual tokens. Accordingly, the authors should ideally report the aforementioned efficiency values for both QART and the baselines.
- Although such a comparison is a major claim of the paper, Table 2 also fails to establish whether input-dependent number of visual tokens is better than a setting with the same number of visual tokens for each input at comparable costs. To exemplify, the text-aware setting has 17.25 visual tokens with 92.01 whereas the setting with 32 visual tokens has 92.08. Though this indeed supports that visual token reduction compared to the 32 token setting is promising, it does not show, e.g. whether the proposed strategy improves over a setting with 17 visual tokens for each input.
- W4: Problems with Ablations and Their Missing Details
- There are several important missing details w.r.t. ablations, notably around the relevance predictor (it having 5 layers, why so and how was this determined), or why the number of visual tokens was cut off at 32 tokens, where this latter decision might adversely affect model performance on vision-centric or OCR-based tasks. Furthermore, the data mixture described in Table 3 has particular ratios w.r.t. each data source, and ideally how the authors came up with it should be clarified.
- More importantly, although the authors partially acknowledge sensitivity to the choice of τ (which I acknowledge is a strength from an honesty perspective), Table 5 in the Appendix shows that different benchmarks have different preferred values. On WinoGround, QART has 19.25 at τ=0.25 and 15.25 at τ=0.20, which is worse than the base SigLIP-2. These results unfortunately bring concerns about the generalizability of the chosen values across the broad settings in which LVLMs are used.
[A] Mao, L., Corona, R., Liang, X., Yan, W., & Tang, Z. (2025). Images are worth variable length of representations. arXiv preprint arXiv:2506.03643.
[B] Hu, W., Dou, Z. Y., Li, L. H., Kamath, A., Peng, N., & Chang, K. W. (2024). Matryoshka query transformer for large vision-language models. Advances in Neural Information Processing Systems, 37, 50168-50188.
[C] Cai, M., Yang, J., Gao, J., & Lee, Y. J. (2025, May). Matryoshka multimodal models. In International Conference on Learning Representations (Vol. 2025, pp. 46254-46272).
[D] Beyer, L., Steiner, A., Pinto, A. S., Kolesnikov, A., Wang, X., Salz, D., ... & Zhai, X. (2024). Paligemma: A versatile 3b vlm for transfer. arXiv preprint arXiv:2407.07726.
[E] Steiner, A., Pinto, A. S., Tschannen, M., Keysers, D., Wang, X., Bitton, Y., ... & Zhai, X. (2024). Paligemma 2: A family of versatile vlms for transfer. arXiv preprint arXiv:2412.03555.
[F] Endo, M., Wang, X., & Yeung-Levy, S. (2025, October). Feather the throttle: Revisiting visual token pruning for vision-language model acceleration. In 2025 IEEE/CVF International Conference on Computer Vision (ICCV) (pp. 22826-22835). IEEE.
[G] Liao, C., Wang, W., Wen, Z., Zheng, X., Wang, Y., He, H., ... & Hu, X. (2026, July). Are we using the right benchmark: An evaluation framework for visual token compression methods. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 4236-4253).
[H] Liu, H., Li, C., Li, Y., & Lee, Y. J. (2024, June). Improved baselines with visual instruction tuning. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 26286-26296). IEEE.
We would like to thank the reviewer for the feedback. We have answered to the questions below.
Visual7W image overlap (Q1) Following the reviewer’s feedback, we investigate the image overlap on Visual7W and identify that there is indeed overlap with QART tokenizer pretraining data. Hence, in order to fairly evaluate QART’s performance, we partition Visual7W images by whether they occurred in QART’s training data and report the results in Table 6 of the main paper. QART obtains 81.85% on unseen versus 82.04% on seen images, while DART obtains 60.12% and 59.68%, respectively. QART's 21.73-point gain on unseen images shows the grounding advantage is not restricted to images seen at pretraining. The full results are shown in Tab. 13 of appendix.
Efficiency Analysis Hardware (Q2) Yes it is the same hardware and full inference path at all batch sizes.