acceptodds
Under review as a conference paper at ICLR 2027

Query-adaptive Recurrent Tokenization

Abstract

Visual information worth extracting from an image and the token budget it warrants depend on the text query. Yet modern vision and vision-language models largely encode images independently of the query and allocate a fixed visual token budget to every input. We introduce the Query-Adaptive Recurrent Tokenizer (QART), a backbone-agnostic module that makes visual representations both query-steerable and budget-adaptive. QART steers the visual representation toward query-relevant evidence while adapting its token budget to the amount of information required. Its key principle is simple. Additional tokens are allocated until the visual features relevant to the query are represented with sufficient fidelity. Starting from frozen encoder features, QART recurrently constructs a compact representation and uses reconstruction quality over the query-relevant visual support to control token allocation. On Visual7W and VQAv2 , QART achieves 81.96% and 76.37% accuracy using only about 29 visual tokens, reducing token counts by 20–25 relative to dense VLM inputs. It further reaches 92.0% P@1 on CORE and 36.3% PR-AUC on PODS, outperforming SteerViT on PODS by 8.4%. Together, these results show that visual representations need not be fixed in either content or capacity and can instead adapt to what each query requires.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Looped transformer, query-adaptive tokenization.

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_xeU8OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

QART builds query-conditioned visual tokens from frozen image features and adds slots recurrently when a query-weighted reconstruction criterion is not met. It reports strong results on grounding and retrieval, preserves 76.37% VQAv2 accuracy with about 29 tokens, and speeds up end-to-end inference only at larger batch sizes.

Critical strengths

The paper separates query-based evidence selection from adaptive token allocation and tests these roles with controlled ablations. The Visual7W comparison and fixed-query Winoground intervention provide useful evidence that query conditioning affects the representation.

Critical weaknesses
  1. The strongest grounding result is obtained after task-specific projector training on Visual7W, while the tokenizer itself is pretrained on Visual Genome and other annotated sources. This is not an unfair compariso, as the baselines receive a similar integration protocol but the result supports compression in a task-adapted VLM more directly than it supports an off-the-shelf tokenizer that transfers without task-specific alignment.

  2. The end-to-end efficiency benefit is batch-dependent: QART runs at 0.87× the dense model's speed at batch size one and reaches 1.53× only at batch size eight. The method is therefore not faster in the low-batch regime that its interactive use cases might suggest.

Questions for authors
  1. Are the Visual7W test images disjoint at the image level from the Visual Genome and other image sources used to pretrain QART and train its projector? Please clarify the split/overlap check; this affects how I interpret the strongest grounding result.
  2. The reported batch-one end-to-end speed is 0.87×. Is this measurement for the same hardware and full inference path as the batch-four/eight measurements, including QART's recurrent token construction?
3 comments

We would like to thank the reviewer for the feedback. We have answered to the questions below.

Visual7W image overlap (Q1) Following the reviewer’s feedback, we investigate the image overlap on Visual7W and identify that there is indeed overlap with QART tokenizer pretraining data. Hence, in order to fairly evaluate QART’s performance, we partition Visual7W images by whether they occurred in QART’s training data and report the results in Table 6 of the main paper. QART obtains 81.85% on unseen versus 82.04% on seen images, while DART obtains 60.12% and 59.68%, respectively. QART's 21.73-point gain on unseen images shows the grounding advantage is not restricted to images seen at pretraining. The full results are shown in Tab. 13 of appendix.


Efficiency Analysis Hardware (Q2) Yes it is the same hardware and full inference path at all batch sizes.

The rebuttal clarifies the setup but does not resolve my main concerns. The overlap analysis does not explicitly cover projector training, the Visual7W results still rely on task-specific adaptation. These limitations materially weaken the transferability and efficiency claims. I therefore maintain my original rating and remain unconvinced that the paper meets the standard for acceptance at ICLR.

We appreciate the reviewer for the prompt response. Regarding the task-specific adaptation of VLM, while we do that training for Visual7W, we have added additional results on several spatial benchmarks like CVBench, EmbSparial, GQA etc. and also add results on generic QA benchmarks without having task-specific adaptation. The results are presented in Tables 3 and 4 of the revised main paper. We would like the reviewer to look through those results and reconsider their rating as that directly assesses the influence of QART on VQA. We also show results on embodied control tasks in. Table 7 of the revised main paper. We would really appreciate it if the reviewer could reconsider their score after going through those results

Reviewer_hJmVOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
2(+1)
Read review
Paper summary

This work proposes an input-dependent visual tokenization method for vision-language understanding. Starting with the observation that visual tokens should ideally vary dependent on the textual task prefix (referred to as "queries" by the authors), the work proposes to use a relevancy predictor and a text-based mechanism to predict the number of visual tokens based on the incoming input. The authors then attempt to demonstrate the efficacy of the proposed approach on two vision-language benchmarks.

Critical strengths

The work has the following two primary strengths:

  • Interesting Dual Perspective on Model Steering and Compression for Efficiency

    • The work approaches the problem of efficient vision-language inference from an interesting perspective on model steering, reporting results both based on steering and efficiency perspectives.
    • Another nicety of the work is that the authors condition the compression based on textual task prefix, which is sound and is a good practice for such a task.
  • Honest and Clear Positioning w.r.t. the Literature and Description of Some Limitations

    • The work has a nicely structured related work section, which compares and contrasts the work in a clear and honest way, going beyond static enumeration of past works, which I appreciated. Although there are some missing works and comparisons, which I discuss under the weaknesses, I believe that the authors did a good job in the narrative and structure of these sections.
    • Likewise, w.r.t. at least some limitations, e.g. around the τ\tau sensitivity, the authors are honest and open.
    • Overall, I genuinely liked the presentation of the work and its narrative. It is well structured and is clear to follow.
Critical weaknesses

The work has the following primary weaknesses:

  • W1: Novelty and Missing Comparisons
    • As it stands, there are several relevant works that the work is not discussing or comparing against, but should have. To exemplify, the authors only cite but do not discuss or compare against Q-DOVE [A], which is an especially relevant work as it also has a similar text-conditioned visual tokenization strategy with a very similar reconstruction strategy where it weights reconstruction toward relevant image regions and producing a variable number of tokens. At the very least, the authors should discuss in detail how their work differs from it, and ideally provide qualitative and quantitative comparisons against it.
    • The work also cites MQT [B], which is another query-based elastic compression method but has no comparisons against it, while failing to cite another similar elastic compression competitor based on pooling altogether, M$^3$ [C]. Given both baselines are training-based elastic compression methods, the authors should ideally empirically compare QART against them.

  • W2: Problems with Experiments, Evaluation Fairness and Breadth
    • For vision-language understanding under compression, the work primarily relies on two benchmarks: VQAv2 and Visual-7W, and there are several issues around both the evaluation fairness and breadth across these settings.
    • For VQAv2, many baselines reported in Table 1 are training-free and/or were initially proposed for CLIP-based LLaVA 1.5, whereas QART requires additional training with dense annotations while utilizing SigLIP-2. This renders the existing comparisons not fully-apples-to-apples. Although I appreciate the inclusion of one such SigLIP-2 comparison in the table, the authors do not report a QART variant with a CLIP-based LLaVA 1.5, matching the original set-up of the baselines. Accordingly, at least including such a setting, clearly marking which baselines are training-free, and including more training-based baselines (e.g. those mentioned under W1), would be a better way to benchmark QART.
    • For Visual7W, given both (i) most, if not all, of these benchmarks omit this benchmark and (ii) all are adapted with a SigLIP-2 backbone, the authors should describe precisely and in detail how were the baseline settings transferred. The Appendix of the work states that all learnable baselines were given an additional 44 more epochs of training, though it is unclear how this training duration was determined and whether a comparable tuning to that of QART was performed for each of the baselines. In any future revisions, this aspect should ideally be clarified in detail.
    • More importantly than the above items, the evaluation presented in the work is very limited for a method aiming to improve the visual tokenization strategy of generalist LVLMs. From the two benchmarks, VQAv2 is known to be a relatively easy benchmark to retain strong performance with low-resolution inputs or smaller number of visual tokens [D, E, F, G] and its training images are included in the standard LLaVA training set-up [B, C, H], making benchmarking there show no indication of zero-shot performance. Furthermore, Visual7W falls short of fully addressing this issue (the precise issue with it is described under the following item). Accordingly, the authors should report results with significantly more and diverse benchmarks, including but not limited to TextVQA, DocVQA, ChartQA, TextCaps and more.
    • Visual7W and RefCOCO are based on the images of the same dataset, MS COCO. The authors should thus clarify precisely whether the Visual7W evaluation images they are reporting the results on were initially present in the training data. On a similar note, the use of RefCOCO during training also blocks the authors from reporting results on that particular benchmark, which is a well-used benchmark for benchmarking LVLMs under visual token compression [F]. Ideally, to test the grounding capabilities of the proposed approach, some comparisons on RefCOCO should be reported and compared against baselines, e.g. against FEATHER [F], after removing RefCOCO data from the training mixture of QART.
    • Finally, both the LVLMs utilized by the authors are based on significantly older LLMs. Ideally, at least some newer backbone should be used for reporting results and under more diverse evaluation settings, such as a member of the newer Qwen/InternVL model families.

  • W3: Concerns with Added Computational Costs
    • Comparing QART and the baselines primarily based on the number of visual tokens is unfair, as it is not established whether they produce the same or similar computational costs, e.g. w.r.t. inference FLOPs, KV cache usage or latency. Although I acknowledge that Figure 4 reports E2E inference results for QART and the base model, the relevance predictor and the additional processing have at least some costs beyond the number of visual tokens. Accordingly, the authors should ideally report the aforementioned efficiency values for both QART and the baselines.
    • Although such a comparison is a major claim of the paper, Table 2 also fails to establish whether input-dependent number of visual tokens is better than a setting with the same number of visual tokens for each input at comparable costs. To exemplify, the text-aware setting has 17.25 visual tokens with 92.01 whereas the setting with 32 visual tokens has 92.08. Though this indeed supports that visual token reduction compared to the 32 token setting is promising, it does not show, e.g. whether the proposed strategy improves over a setting with 17 visual tokens for each input.

  • W4: Problems with Ablations and Their Missing Details
    • There are several important missing details w.r.t. ablations, notably around the relevance predictor (it having 55 layers, why so and how was this determined), or why the number of visual tokens was cut off at 3232 tokens, where this latter decision might adversely affect model performance on vision-centric or OCR-based tasks. Furthermore, the data mixture described in Table 3 has particular ratios w.r.t. each data source, and ideally how the authors came up with it should be clarified.
    • More importantly, although the authors partially acknowledge sensitivity to the choice of τ\tau (which I acknowledge is a strength from an honesty perspective), Table 5 in the Appendix shows that different benchmarks have different preferred values. On WinoGround, QART has 19.2519.25 at τ=0.25\tau=0.25 and 15.2515.25 at τ=0.20\tau=0.20, which is worse than the base SigLIP-2. These results unfortunately bring concerns about the generalizability of the chosen values across the broad settings in which LVLMs are used.

[A] Mao, L., Corona, R., Liang, X., Yan, W., & Tang, Z. (2025). Images are worth variable length of representations. arXiv preprint arXiv:2506.03643.

[B] Hu, W., Dou, Z. Y., Li, L. H., Kamath, A., Peng, N., & Chang, K. W. (2024). Matryoshka query transformer for large vision-language models. Advances in Neural Information Processing Systems, 37, 50168-50188.

[C] Cai, M., Yang, J., Gao, J., & Lee, Y. J. (2025, May). Matryoshka multimodal models. In International Conference on Learning Representations (Vol. 2025, pp. 46254-46272).

[D] Beyer, L., Steiner, A., Pinto, A. S., Kolesnikov, A., Wang, X., Salz, D., ... & Zhai, X. (2024). Paligemma: A versatile 3b vlm for transfer. arXiv preprint arXiv:2407.07726.

[E] Steiner, A., Pinto, A. S., Tschannen, M., Keysers, D., Wang, X., Bitton, Y., ... & Zhai, X. (2024). Paligemma 2: A family of versatile vlms for transfer. arXiv preprint arXiv:2412.03555.

[F] Endo, M., Wang, X., & Yeung-Levy, S. (2025, October). Feather the throttle: Revisiting visual token pruning for vision-language model acceleration. In 2025 IEEE/CVF International Conference on Computer Vision (ICCV) (pp. 22826-22835). IEEE.

[G] Liao, C., Wang, W., Wen, Z., Zheng, X., Wang, Y., He, H., ... & Hu, X. (2026, July). Are we using the right benchmark: An evaluation framework for visual token compression methods. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 4236-4253).

[H] Liu, H., Li, C., Li, Y., & Lee, Y. J. (2024, June). Improved baselines with visual instruction tuning. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 26286-26296). IEEE.

Questions for authors
  • Q1: Could the authors clarify how QART differs from Q-DOVE and the other existing works mentioned under W1, and clarify why the work does not compare against them?
  • Q2: Could the authors report a QART variant with the CLIP-based LLaVA 1.5 backbone, and describe precisely how the Visual7W baselines were adapted to SigLIP-2 and tuned?
  • Q3: Could the authors report inference FLOPs, KV Cache usage, peak memory and end-to-end inference or throughput for QART and the baselines, as well as the number of visual tokens?
  • Q4: Could the authors compare the query-aware strategy against a setting with the same number of visual tokens for each input at comparable computational costs, rather than only against the 32-token setting in Table 2?
  • Q5: Could the authors report results on RefCOCO after removing all RefCOCO data from the training mixture, including a comparison against FEATHER? Could they also clarify whether the Visual7W evaluation images were present during training?
  • Q6: Could the authors report results with more vision-centric benchmarks and different numbers of visual tokens, and clarify how tau was chosen for each reported benchmark?
  • Q7: Could the authors clarify the choice of 5 layers, the 32-token cut-off and the data mixture of Table 3?

That being said, I am open to discussion by authors and would like to encourage them to address my concerns regardless of my initial score.

4 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer hJmV

We thank the reviewer for the detailed feedback. While we have expanded the QA benchmarks, added new baselines and addressed all the concerns of the reviewer to the best of our possibility, we would like to clarify the central claim: QART is a general-purpose visual tokenizer designed to be both query-steerable (what evidence is represented) and budget-adaptive (how much capacity is allocated). VQA is one downstream test of representation utility, not QART's sole objective. We evaluate these properties through retrieval, compositional matching, grounding, perception, as well as VQA. We have also added a new task to evaluate embodied control.


Matched backbone and wider evaluation (Q2, Q6) We train CLIP-based QART using CLIP ViT-L/14, retaining the SigLIP2 tokenizer's training mixture and hyperparameters. For CLIP-based QA, we follow LLaVA's recipe of one alignment epoch and one instruction-SFT epoch using LLaVA data. As shown in Table 3 (Page 10), against nine compression baselines, including MQT and M³, QART leads on 7/10 spatial/compositional metrics: VSR 61.13% (59.82% best compressed), EmbSpatial 46.17% using 18.47 tokens (46.24% dense), and CVBench-2D 60.30% (59.12% best compressed). Additional QART results on General QA are shown in Table 4. These results directly address backbone comparability and evaluation breadth. For all the experiments, the relevance threshold is selected per benchmark on held-out validation data. However, extensive hyperparameter tuning has not been conducted.

For Visual7W, all methods instead use SigLIP2-based LLaVA: one Visual Genome alignment epoch and six Visual7W projector-training epochs, with four additional epochs for learnable compression baselines after incorporating those compression techniques. We clarify this distinct protocol in the Appendix A.3 of the main paper.


Visual7W image overlap and RefCOCO (Q5) Following the reviewer’s feedback, we investigate the image overlap on Visual7W and identify that there is indeed overlap with QART tokenizer pretraining data. Hence, in order to fairly evaluate QART’s performance, we partition Visual7W images by whether they occurred in QART’s training data and report the results in Table 6 of the main paper. QART obtains 81.85% on unseen versus 82.04% on seen images, while DART obtains 60.12% and 59.68%, respectively. QART's 21.73-point gain on unseen images shows the grounding advantage is not restricted to images seen at pretraining. The full results are shown in Tab. 13 of appendix. A strictly RefCOCO-disjoint evaluation would require excluding RefCOCO-family annotations and retraining QART; This has neither been shown by token compression methods [1,2,3] nor by steerability methods [4]. Hence, we think it is unfair to ask us for such evaluations, and it is also infeasible within the rebuttal period given the resource constraint.


Efficiency (Q3) All new efficiency measurements use Visual7W with SigLIP2-based LLaVA. As shown in Table 8, relative to dense LLaVA, QART reduces TFLOPs 12.82→3.81 (70.3%), KV-cache 450.4→100.8 MB (77.6%), and batch-eight prefill latency 456.9→95.0 ms (4.81×), including its recurrent construction cost in end-to-end throughput: 15.0→40.2 questions/s (2.68×). At nearly identical batch-eight throughput, QART reaches 81.96% Visual7W accuracy versus DART's 59.86% (40.2 vs. 40.0 questions/s). QART incurs more single-example overhead than DART.


Token Allocation (Q4) On CORE, query-aware growth/halting achieves 92.01% P@1 with 17.25 tokens and 4.25 rounds, versus 92.08% with all 32 tokens and six rounds: 46% fewer active tokens and 29% fewer rounds with comparable accuracy. Separately, a fixed-budget relevance top-k control uses QART's own relevance map and a retrained projector; at approximately 29 tokens, it reaches 49.46% on Visual7W versus QART's 81.96%, while 128 retained patches reach 58.98%.


[1] Li et al., "OccamToken: Efficient VLM Inference with Training-Free and Budget-Adaptive Token Pruning", arXiv'26 [2] Hu et al., "Matryoshka Query Transformer for Large Vision-Language Models", ICLR'24 [3] Takezoe et al., "LearnPruner: Rethinking Attention-based Token Pruning in Vision Language Models", ICLR'26 [4] Ruthardt et al., "Steerable Visual Representations", ECCV'26

AuthorsOpenReview ↗

Rebuttal to reviewer hJmV continued

Novelty, Q-DOVE, and generality beyond VQA (Q1)

Q-DOVE learns query-conditioned dynamic allocation implicitly through learned EOS termination and region-weighted image reconstruction loss during training. In contrast, QART evaluates reconstruction coverage of query-relevant pretrained visual features at inference, explicitly using feedback after recurrent refinement to add slots or halt; relevance prediction and slot initialization also depend on the query. Q-DOVE's Table 3 reports higher accuracy with a null query than the actual question on all eight benchmarks at the same 256-token budget (e.g., VQAv2 55.0% vs. 53.9%), indicating that adding the query does not improve performance. By comparison, replacing QART's specific query with fixed conditioning decreases Winoground Group accuracy from 19.25% to 12.00%, supporting genuine query steerability. MQT and M³ offer flexible nested budgets, but do not use QART's reconstruction-driven per-query stopping rule. We now compare against both under the CLIP-based setting; Q-DOVE has substantially different vision-tokenization/training architecture, precluding a controlled comparison of published accuracy numbers.

Critically, we see that our method extends to tasks beyond VQA. Specifically, QART generalizes to action prediction on LIBERO-Plus Goal dataset: a robot action prediction benchmark which contains distractor objects and displaced objects. A downstream transformer policy with frozen QART features reaches 52.6% success with task-specific instructions versus 40.7% with generic conditioning (+11.9 points). It also exceeds the reported π0 result (44.7%). This demonstrates that query steerability matters for embodied action, not only VQA. The results are shown in Table 7.


Architecture and mixture (Q7) The training data mixture was decided to balance precise grounding supervision (50% referring-expression data) with richer attributes and relations (35% Visual Genome) and outdoor scenes (15% Mapillary). Source-weighted sampling prevents the largest dataset from dominating.These proportions were set during iterative development.

Regarding number of layers and hyperparameters, they were not optimized or tuned heavily. For the number of layers, 3 layers and 5 layers were initially evaluated just on the refCOCO data and we found out that 5-layer relevance module improved performance while not increasing training time significantly. Regarding 6 rounds and 32 tokens, those were choices made to work within the constrained compute resources that were in our disposal. While we acknowledge that larger token counts could benefit general reasoning, we could not evaluate those due to compute and time constraints. The hyperparameters such as ηgrow\eta_{grow} and ηhalt\eta_{halt} were also decided during early development, without further refinement. However, we do see that tuning these hyperparameters results in much lesser tokens on VQAv2 (from 29.45 to 17.52) while only reducing accuracy by 0.68% (76.37 vs 75.69). This tells us that meticulous tuning of hyperparameters could further increase the efficacy of QART.


Reviewer_hJmVOpenReview ↗

Response to the Authors (1/2)

I would like to thank the authors for their efforts in their detailed response. Below, I leave my comments on each item I raised:

  • On Q1 and Comparisons Against Baselines:

    • On Q-Dove: I understand that there are some differences between the proposed approach and Q-Dove. However, I do not believe that omitting it altogether from the main paper is reasonable as the reconstruction aspect of the work has already been realized with that work (w/ or w/o text prefix is another side, I understand, but that should not limit the authors from acknowledging this work). Furthermore, the work itself already describes ALIT as the closest architectural precedent w.r.t. recurrent, reconstruction-based variable-length tokenization. It then follows that the novelty aspect of the work is then limited to a text prefix-based instantiation of existing reconstruction-based works on compression in the domain. In my opinion, an honest acknowledgement of this aspect would be better for the narrative of the work.
    • Regarding quantitative comparisons against Q-Dove, could the authors precisely detail what prevents them from having at least some comparison? I am not arguing that the authors must have that comparison, but merely mentioning this for the sake of reader clarity as I believe that the work would benefit from mentioning and detailing this clearly in its main text.
    • On comparisons with MQT and M^3, there seems to be a disparity w.r.t. SEED-Bench numbers that affects not only the two models but also the LLaVA 1.5 baseline which made me confused initially about whether something went wrong with the M^3 reproduction at 36 tokens. Upon realizing that all models seem to suffer more or less equally compared to the paper-published numbers, I believe that this might be pointing toward an issue with the evaluation set-up (e.g. maybe only considering a subset of SEED-Bench instead of the full setting? Is this a repeated choice across other benchmarks too?). I believe that some clarification on what changed here would be valuable.
    • I acknowledge the authors have also presented results with the LIBERO-Plus Goal dataset. However, I do not see any specifications provided for this dataset, the baselines used in Table 7 or how QART was utilized here. Furthermore, Table 7 in the main paper only includes π0 and OpenVLA, whereas Table 14 in the Appendix shows that QART is behind several baselines, e.g. OpenVLA-OFT, RIPT-VLA, NORA and the fine-tuned LIBERO-Plus policy. Accordingly, the claim of outperforming π0 in the main text does not fully reflect these results. From what I can tell as a reader, this under-specification seems to be a problem of the work (e.g. also the weakness I raised w.r.t. ablations) and I genuinely believe that the true contributions of the work could be better perceived by an improved narrative.
    • Accordingly, I can only say that my concern on this item has been partially addressed.
  • On Q2 and Q6:

    • I acknowledge the results added by the authors. The CV-Bench numbers are indeed promising. Their inclusion supports the hypotheses of the work. However, I wonder why the authors omitted the benchmarks listed in my review initially (e.g. TextVQA, DocVQA, ChartQA), which are used a lot more often for LLaVA 1.5 backbones than the benchmarks chosen by the authors.
    • Furthermore, Table 4 does not include any of the compression baselines, only QART and the dense model, which makes it unable to support comparisons against the baselines.
    • The “better at 7/10 metrics” claim also seems to count the four CV-Bench subsets separately as if they were separate benchmarks. Per benchmark, however, QART leads on VSR, EmbSpatial and CV-Bench, but is behind the baselines on SEED and GQA (although admittedly slightly)
    • Accordingly, my concern on this item has only been partially addressed.
  • On Q3:

    • I thank the authors for Table 8. However, I would like note that at bs=1, QART has the lowest throughput among all compression methods (with 14.3 vs. 17.0-19.1), and at bs=8, it is behind DivPrune and LearnPruner.
    • Furthermore, Table 8 shows a 1.21x speed-up over the dense model at batch size 1, whereas the original submission reported 0.87x at batch size 1. Could the authors clarify this difference?
    • On a similar note, reducing the number of visual tokens on VQAv2 from 29.45 to 17.52 only improves the throughput from 1.53x to 1.58x, which suggests that the added computational cost of QART itself is actually not negligible.
Reviewer_hJmVOpenReview ↗

Response to the Authors (2/2)

  • On Q4:

    • Unfortunately, the response does not address my question, as it seem to simply repeat the CORE comparison against the 32 token setting. My question was whether the proposed strategy improves over a setting with around 17 visual tokens for each input at comparable computational costs.
    • Furthermore, the top-k relevance setting reaches 49.46% at around 29 tokens, which is below the Random baseline in Table 13 (51.16%). Could the authors clarify how this setting was configured?
    • Accordingly, I cannot say that my concern on this item has been addressed.
  • On Q5:

    • I thank the authors for the seen/unseen partition. However, Table 13 and Section 6 describe it w.r.t. the Visual Genome alignment data, whereas the response states that it is w.r.t. the tokenizer pretraining data. Could the authors clarify which one is the case, and whether the COCO images of RefCOCO are also included?
    • On RefCOCO, I understand the compute constraints. However, I do not agree that this request is unfair. Given RefCOCO is part of the training mixture of QART, an evaluation after removing it would be the natural way to test the grounding claims of the work. Still, I am not fully penalizing the work for missing these and these did not take a major part in my final opinion on the work.
  • On Q7:

    • I thank the authors for the clarifications. Ideally, these should be included in the Appendix in any future revisions, along with some explanation on what "These proportions were set during iterative development." precisely entails.

Overall, I acknowledge that I raised quite a few questions and weaknesses and would like to thank the authors for their significant efforts in attempting to address them. Unfortunately, however, the primary claims and conclusions of the work along with its clear positioning is still missing from its main text, along with the aforementioned partially addressed or unaddressed issues. I do not believe that the work is of bad/low quality and I believe that it could be a very strong submission, I just do not think that its current version is at that stage yet. Accordingly, I will be keeping my score as is for now.

Reviewer_gWVXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes QART, a visual tokenizer on top of frozen SigLIP2 features that makes the visual representation depend on the text query in both content and length. A cross-attention relevance head scores each patch against the query and keeps the patches above a threshold τ\tau. Starting from two query-initialized slots, a weight-shared three-layer Transformer recurrently refines the slots over these patches, adding six slots per round (up to 32 slots and six rounds). A shared decoder reconstructs the feature map after each round; slots are added while fewer than ηgrow\eta_{\text{grow}} of the relevant patches are reconstructed within error δ\delta, and recurrence stops once this coverage exceeds ηhalt\eta_{\text{halt}}. The tokenizer is pretrained for 25 epochs on about 2.1M referring-expression and region annotations (RefCOCO family, Mapillary Vistas, Visual Genome), with dense relevance targets.

Validation covers representation steering (CORE P@1 68.5% → 92.0% over native SigLIP2; GeneCIS zero-shot; PODS +8.4 PR-AUC over SteerViT), compositional matching (Winoground, with a fixed-query control), and VLM use with a frozen LLaVA-1.5-7B and a trained projector. On VQAv2 it reaches 76.37% with about 29 tokens (93.8% of the dense SigLIP2 model), and on Visual7W pointing it beats re-run pruning baselines by 22 points. Ablations separate query-steered support from query-aware budgeting, and the appendix covers a DINOv2 backbone and threshold sensitivity.

Critical strengths

S1. A single, testable principle linking content and capacity, with ablations that isolate both roles. QART ties the token count to an explicit adequacy criterion: keep adding slots until the query-relevant features are reconstructed well enough. The CORE ablation shows the query plays two separable roles. Removing query-steered support drops P@1 from 92.0 to 79.7, while removing query-aware budgeting leaves accuracy unchanged (92.08) but raises the budget from 17.25 to 32 slots and from 4.25 to 6 rounds. The Winoground fixed-query control (Group 19.25 → 12.00) also shows the representation follows the query rather than only the image. This is the paper's most convincing contribution.

S2. Large gains where the query actually matters. Because the comparison holds the frozen encoder fixed, the CORE improvement (68.5% → 92.0% P@1) can be attributed to query-conditioned construction rather than a stronger backbone. The gains also transfer zero-shot to GeneCIS and to PODS, where a coarse prompt still preserves instance-level detail (+8.4 PR-AUC over SteerViT). On Visual7W pointing, with all baselines re-run on the same SigLIP2 base, QART beats DART by 22 points at about 29 tokens.

Critical weaknesses

W1. The adaptive budget and efficiency claims are weakly supported in the VLM setting, and the recurrence is not compared with simpler alternatives. On VQAv2 QART uses 29.45 of 32 slots on average, so the token budget is close to fixed. The headline 20–25× reduction is relative to 729 dense patches, not to a fixed 32-token resampler. End-to-end inference is slower than the dense model at batch size 1 (0.87×) and only 1.53× faster at batch size 8. Evidence that adaptive allocation saves tokens comes almost entirely from the CORE ablation.

The paper also does not compare against the obvious simpler designs that use the same relevance head: (i) feeding the top-kk relevance-ranked patches directly to the LLM, and (ii) a fixed 32-query resampler over VAV_{\mathcal{A}}. Without these, it is unclear whether the gains come from the recurrent reconstruction-guided allocation or from the relevance head trained on dense region annotations.

W2. Narrow VLM evaluation and unequal comparison conditions. VQAv2 is the only general VQA benchmark. There is no GQA, TextVQA, MMBench, or POPE, and no high-resolution VLM. On VQAv2 the pruning baselines use a different backbone and copied numbers, and under the reported relative-accuracy metric QART (93.8%) is below OccamToken (96.2%), QMoP (94.4%), and LearnPruner (94.3%). On Visual7W, QART's tokenizer is pretrained on Visual Genome region annotations, while the compared pruning methods are training-free. Visual7W images come from Visual Genome and COCO, and the paper does not state whether pretraining images overlap with the test images. As a result, the strongest VLM result may partly reflect extra supervision rather than the tokenizer design.

Questions for authors
  1. Simpler baselines with the same relevance head. How do (i) top-kk relevance-ranked patches fed directly to LLaVA, and (ii) a fixed 32-query resampler over VAV_{\mathcal{A}}, perform on VQAv2, Visual7W, and CORE at matched token counts? This is the key test of whether recurrent allocation is needed.
  2. Adaptive budget in the VLM setting. What is the distribution of active slots on VQAv2 and Visual7W (not only the mean), and what accuracy and latency do you get if ηgrow\eta_{\text{grow}}, ηhalt\eta_{\text{halt}} are tuned so that the mean is well below 32 (e.g., 16)?
  3. Data overlap on Visual7W. Do any Visual7W test images or their region annotations appear in the Visual Genome or COCO portions of the tokenizer pretraining mixture? If so, what is the result after removing them?
  4. Threshold selection. How were τ\tau (0.05 for VQAv2, 0.25 elsewhere), δ\delta, ηgrow\eta_{\text{grow}}, and ηhalt\eta_{\text{halt}} selected? Were any chosen on the reported test splits?
1 comment
AuthorsOpenReview ↗

Rebuttal to Reviewer gWVX

We would like to thank the reviewer for their invaluable feedback. We have addressed all the concerns raised to the best of our ability.

More results on VLM Benchmarks (W2) To evaluate on a wider set of QA benchmarks, we train a CLIP-based QART variant using CLIP-L/14 as the visual encoder and CLIPText as the text encoder. Our QART CLIP variant upon integration with LLaVA-v1.5-7B leads on 7/10 spatial/compositional metrics against nine token compression baselines: VSR 61.13% (59.82% best compressed), EmbSpatial 46.17% using 18.47 tokens (46.24% dense), and CVBench-2D 60.30% (59.12% best compressed). Additional QART results on General QA are shown in Table 4 of the revised main paper.


Visual7W data overlap (Q3) We would like to thank the reviewer for this question. Following the reviewer’s feedback, we investigate the image overlap on Visual7W and identify that there is indeed overlap with QART tokenizer pretraining data. Hence, in order to fairly evaluate QART’s performance, we partition Visual7W images by whether they occurred in QART’s training data and report the results in Table 7 of the main paper. QART obtains 81.85% on unseen versus 82.04% on seen images (0.19-point difference), while DART obtains 60.12% and 59.68%, respectively. QART's 21.73-point gain on unseen images shows the grounding advantage is not restricted to images seen at alignment. The full results are shown in Tab. 13 of appendix.


Token Analysis (Q2) Again, we would like to really thank the reviewer for this question! Given your advice and feedback, we actually tried to assess the impact of tuning ηgrow\eta_{grow}, ηhalt\eta_{halt} and τ\tau on the token distribution and query understanding of QART. Specifically, we analyzed the token distribution VQAv2 using the default setting mentioned in the main paper, and the tuned setting, and we also assess the correlation of token count with (image, query) complexity. The results are shown in Fig. 8 and Appendix D.1. From the results, we see that the token distribution is heavily skewed towards consuming all 32 tokens (73% of the questions) in the default setting. However, when we mildly tune the hyperparameters, we see the average token count to drop from 29.45 to 17.52 while only changing accuracy from 76.37% to 75.69% and improving throughput from 1.53x to 1.58x wrt dense LLaVA. Further, we also see that in the tuned policy, the tokens are correlated to the complexity of query-relevant region in the image, while its less correlated to the entire image, showing that our tokens focus on query-relevant images. This experiment actually tells us that meticulous tuning of hyperparameters could further improve our tokenizer making it a more impactful solution. We did not prioritize extensive task-specific hyperparameter optimization in our initial experiments because the primary objective of this work is to introduce a general-purpose query-adaptive visual tokenizer that addresses both the content and capacity dimensions of visual tokenization, rather than developing a highly optimized, task-specific solution for VQA. However, we acknowledge that we have not tuned it enough but we believe that tuning will only improve our method. We would like to once again thank the reviewer as we only focused on this experiment because of their feedback.


Top-k relevance and fixed 32-token resampler (Q3):- We train a fixed-budget relevance top-k control that uses QART's own relevance map and a retrained projector; at approximately 29 tokens, it reaches 49.46% on Visual7W versus QART's 81.96%, while 128 retained patches reach 58.98%. Similarly on VQAv2, budget-matched top-k relevance SigLIP2 patches reaches 63.4% vs QART’s 76.37%. These comparisons show budget savings relative to full-capacity QART and the value of constructed tokens over fixed top-k selection; they do not independently isolate adaptive allocation from representation construction at equal mean budgets. Additionally, we also trained a fixed 32-token resampler on VQAv2 which reached 70.08%, in comparison to 76.37% of QART. This highlights the significance of QART over simpler methods.


Threshold Selection (Q4):- We select our threshold τ\tau using a held-out set for every benchmark.


From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.