acceptodds
Under review as a conference paper at ICLR 2027

RUTA: Principled Visual Token Allocation via Rate-Utility Optimization

Abstract

High-resolution images and long videos provide vision-language models with rich context for multimodal reasoning and fine-grained perception, but the resulting long visual token sequences make large language model-side computation and memory costly. Existing visual token reducers often operate at prescribed rates, while recent methods adapt token counts across inputs using method-specific learned thresholds or importance predictors. We introduce RUTA, a principled ate-tility oken llocation method that performs pre-LLM reduction by jointly learning which tokens to retain and how many to allocate to each image-query pair. RUTA constructs query-conditioned candidate tokens and predicts a retention probability for each candidate. During training, these probabilities parameterize independent Bernoulli gates, while their sum provides a differentiable training-time estimate of the token count for each pair. Retained tokens serve as anchors that aggregate information from non-retained tokens according to semantic affinity and spatial proximity. RUTA is optimized with a penalized rate-utility objective that balances downstream task loss against expected token usage. Averaged across five benchmarks and measured relative to each backbone's full-token baseline, RUTA uses only and of visual tokens while preserving and of task performance on LLaVA-NeXT and Qwen3-VL, respectively.

Primary area:
foundation or frontier models, including LLMs
Keywords:
vision-language models, visual token reduction, adaptive token allocation, rate-utility optimization, efficient inference

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_qwhpOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes RUTA, a learning-based pre-LLM visual token reduction framework designed to jointly determine which visual tokens to retain and how many tokens to allocate for each image-query pair. Specifically, RUTA first identifies query-relevant regions via a query-conditioned localizer, which are then cropped, resized, and re-encoded into candidate tokens. Next, it predicts a retention probability for each candidate token, using the sum of these probabilities as a differentiable estimate of the expected token count. During training, token selection is modeled via independent Bernoulli gates coupled with a straight-through estimator (STE), whereas at inference time, it switches to a deterministic adaptive top-K selection determined by the expected token rate. Non-retained candidate tokens are aggregated into the retained anchor tokens based on semantic affinity and spatial proximity. The entire pipeline is jointly optimized using a downstream task loss and a penalty on expected token usage. The authors evaluate RUTA on LLaVA-NeXT-7B and Qwen3-VL-8B across five visual question answering (VQA) benchmarks, reporting performance gains under low token budgets and providing ablation studies on rate allocation, localizer choices, and anchor merging strategies.

Critical strengths

S1.Problem Formulation and Motivation: The paper formulates visual token reduction as a rate-utility optimization problem inspired by rate-distortion theory. It uses the sum of token retention probabilities as a differentiable estimate of the expected token count and penalizes this estimate to control the trade-off between task performance and token usage. This formulation enables the model to adaptively allocate tokens according to the visual evidence requirements of different image-query pairs, providing a unified optimization framework for mitigating the over-allocation of tokens to simple queries and the under-allocation of tokens to more demanding ones in fixed-budget methods.

S2.Architectural Synergy and Information Preservation: The paper combines query-conditioned region-of-interest (ROI) localization and re-encoding with anchor-based token merging guided by semantic affinity and spatial proximity. This design concentrates visual resolution on query-relevant regions while softly aggregating information from non-anchor tokens into retained anchors. The reported results demonstrate that RUTA maintains competitive task performance at extremely low token budgets, while the merging ablations provide supporting evidence for the benefits of combining semantic and spatial information during aggregation.

S3.Targeted Ablation Studies: The paper conducts informative ablations using fixed-rate, shuffled-rate, and adaptive-rate controls. By comparing these strategies at approximately the same average token budget, the experiments help distinguish the benefits of varying token allocations from those of assigning budgets appropriately to individual inputs. Furthermore, comparisons among anchor-only retention, uniform merging, spatial routing, semantic routing, and the full RUTA method provide targeted evidence regarding the role of non-anchor information aggregation and the contributions of semantic and spatial affinity.

Critical weaknesses

W1. Ambiguous token budget accounting undermines the credibility of the matched-rate comparisons. Table 1 defines "Tok" as the number of visual tokens averaged over samples and LLM layers (consistent with the layer-wise average formulation in Sec. 3.1). However, the reported numbers for in-LLM baselines contradict this definition. For instance, on VQAv2 with LLaVA-NeXT-7B (L=32 layers, N=2,233 reference tokens), an in-LLM reducer that processes full visual tokens in the first m layers before pruning has a strict mathematical lower bound of mN/L >= 69.8 layer-averaged tokens (and >= 139.6 for FastV's default m=2). Yet, Table 1 reports 42.0 tokens for FastV. This discrepancy suggests that either pre-pruning layers were excluded from the average, or different input preprocessing/resolutions were used across baselines, calling into question whether the comparisons in Table 1 and Fig. 3 are genuinely rate-matched.

W2. Unclear gradient pathways in the claimed joint optimization hinder reproducibility. The paper claims to jointly optimize the candidate constructor (updating the localizer's grounding heads) and the rate-controlled compressor via a single end-to-end objective (Eq. 14). However, two critical gradient pathways are unexplained: (1) In Eq. (10)-(13), the straight-through estimator (STE) gates are only applied to the retained anchors in the final output sequence, while non-anchors (xi_j = 0) are indexed out without gate participation in the merging weights (Eq. 12), leaving it unclear how non-anchor retention probabilities receive task gradients to become anchors. (2) Candidate construction (Eq. 4-6) relies on discrete phrase parsing, confidence thresholding, box union, and image cropping, with no explanation of how gradients from the downstream task or rate loss backpropagate through these non-differentiable operations to update the localizer parameters.

W3. Entangled algorithmic gains and limited end-to-end system efficiency advantages. First, while RUTA is fine-tuned end-to-end (including the VLM projector, merging layers, and an external 232M-parameter localizer), most baselines in Table 1 are training-free. Meanwhile, closely related trainable and adaptive reducers discussed in Sec. 2.2 (e.g., CROP, ATP-LLaVA, GlimpsePrune) are omitted from the empirical comparisons, making it difficult to isolate the incremental value of the proposed rate-utility allocation from the benefits of region cropping and task-specific fine-tuning. Second, the system efficiency evaluation in Appendix Table 2 only compares against the unreduced reference rather than competing reducers. On Qwen3-VL-8B, reducing visual tokens by 94.8% (1,337.5 -> 69.3) yields only a 1.12x end-to-end speedup while increasing peak GPU memory (21.0 GB -> 21.7 GB) and dropping accuracy by 9.9 points (83.0% -> 73.1%), which weakens the practical system-efficiency claim.

Questions for authors

Q1. Clarification on token budget accounting and matching protocol in Table 1: How is "Tok" computed for in-LLM baselines such as FastV and SparseVLM? Does it include the unpruned early LLM layers, and do all baselines share the exact same input resolution and multi-view preprocessing? Please explain how FastV achieves 42.0 tokens on LLaVA-NeXT-7B given the layer-average lower bound of mN/L, and clarify how a consistent accounting metric affects the rate-matched rankings.

Q2. Clarification on gradient propagation for joint training: Could you provide pseudocode or a computation graph clarifying: (a) how task gradients backpropagate to the retention probabilities of non-selected tokens (xi_j = 0) and through the anchor-set normalization in Eq. (12)-(13)? and (b) how the localizer's grounding heads are updated across the discrete thresholding, box union, and cropping operations in Eq. (4)-(6) (e.g., whether auxiliary detection losses or differentiable ROI operators are used)?

5 comments

We thank the reviewer for the careful assessment and for highlighting the token-accounting and gradient-flow questions. We clarify these points and the comparison protocol below.


W1 & Q1: Clarification on token budget accounting and matching protocol

Thank you for identifying this inconsistency. Tok denotes the number of visual tokens averaged over samples and LLM layers for in-LLM baselines. However, the caption omitted the markers identifying FastV’s Tok as its retained count after two unpruned layers, denoted here by tFt_{\mathrm F}, and SparseVLM’s Tok as its average count over the remaining layers, denoted by tSt_{\mathrm S}. For both methods, we preserve these unpruned layers to avoid severe accuracy degradation from premature pruning. Most of baselines use the reference model’s input resolution, and multi-view preprocessing within LLaVA-NeXT. However, there are exceptions, such as FastVLM, which modifies the vision encoder and input resolution to improve efficiency.

Specifically, in our SparseVLM experiments, pruning occurs after zero-indexed layers [1, 6, 15]. Let NN denote the input visual-token count, LL the decoder depth, and rjr_j the count after each pruning stage, including recycled tokens. Then tS=5r1+9r2+(L−16)r3L−2.t_S=\frac{5r_1+9r_2+(L-16)r_3}{L-2}. The corrected all-layer token counts, including the two unpruned layers, are $ \mathrm{Tok}_{\mathrm{FastV}}^{\mathrm{corr}} =\frac{2N+(L-2)t_{\mathrm{F}}}{L}, \mathrm{Tok}_{\mathrm{SparseVLM}}^{\mathrm{corr}} =\frac{2N+(L-2)t_{\mathrm{S}}}{L}. $ These quantities are averaged over samples. For example, on VQAv2, FastV’s corrected counts are approximately 178.9 on LLaVA-NeXT and 102.8 on Qwen3-VL; SparseVLM’s are 179.0 and 103.0, respectively. We will add the missing markers in the caption of all affected tables and figures.

RUTA retains the highest mean accuracy on both backbones, while consistent accounting assigns larger token budgets to FastV and SparseVLM. Its advantage at the evaluated operating points therefore remains supported.


W2 & Q2: Clarification on gradient propagation for joint training

Non-selected tokens and anchor normalization (Eqs. 10–13). The training implementation includes an STE factor omitted from Eq. (13). For simplicity, the expression for Δzi\Delta z_i in Eq. (13) omitted the factor 1−ξ~j1-\tilde{\xi}_j. The full expression is

\frac{\sum_{j\in\bar{A}}w_{i,j}(1-\tilde{\xi}_j)W_{\mathrm{value}}z_j} {\max\\!\left(1,\sum_{j\in\bar{A}}w_{i,j}\right)}, \qquad i\in A.$$ For non-selected tokens, $1-\tilde{\xi}_j=1$ in the forward pass, preserving the stated aggregation. Its Binary-Concrete backward relaxation provides **task gradients to their retention probabilities** $\rho_j$, alongside the rate gradient from Eq. (9). Gradients also propagate through the routing softmax and aggregation denominator, with the sampled anchor and non-anchor sets held fixed. **Localizer optimization (Eqs. 4–6).** Threshold-based indexing preserves the computation graph of selected box coordinates. Coordinate-wise minima/maxima construct the enclosing box, and bilinear `grid_sample` implements differentiable cropping and resizing. The frozen vision encoder preserves gradients with respect to its input, giving the backward path $$\ell_{\mathrm{task}}+\lambda_{\mathrm{rate}}\widehat R_\phi \\;\longrightarrow\ Z \\;\longrightarrow\ \text{resampled image} \\;\longrightarrow\ b_{\mathrm{roi}} \\;\longrightarrow\ \text{selected box coordinates} \\;\longrightarrow\ \varphi.$$ Thus, the downstream objective updates the localizer through continuous coordinates conditional on the selected boxes, **without auxiliary detection losses or additional downstream box annotations**. We will correct Eq. (10) to specify the Binary-Concrete backward relaxation, restore the omitted factor in Eq. (13), and make the gradient flow during training more explicit in the revision.

W3. Entangled algorithmic gains and limited end-to-end system efficiency advantages.

Comparison fairness and related methods. Comparisons with training-free reducers test whether RUTA’s training improves inference-time accuracy–compression trade-offs. To address supervision differences, FastVLM and TwigVLM use the same training splits and answer labels as RUTA. We also compare CROP (localization) [A] and GlimpsePrune (adaptive pruning) [B] on TextVQA with Qwen3-VL-8B, preserving their original objectives. GlimpsePrune additionally uses region supervision, whereas RUTA requires only task answers. As shown in Table R7, RUTA outperforms both closely related reducers at comparable low token budgets, complementing the task-supervised comparisons in Table 1.

Table R7. Comparison with CROP-PLC and GlimpsePrune on Qwen3-VL-8B/TextVQA.

MethodAccuracy (%) ↑All-layer mean visual tokens ↓
Reference82.31,337
CROP-PLC65.968.0
GlimpsePrune64.475.3
RUTA73.170.4

 

Contribution of adaptive allocation. Fig. 4(c) fixes the trained model and localization pipeline. At approximately 70 tokens, adaptive allocation improves accuracy by 1.7 points over fixed allocation and 2.0 points over shuffled allocation, which preserves the budget distribution. These controls isolate the benefit of assigning capacity to the appropriate input beyond shared cropping and fine-tuning.

 

Full-pipeline efficiency. We compare RUTA with the two highest-accuracy competing TextVQA reducers on each backbone at the Table 1 operating points (Table R8). On LLaVA-NeXT, RUTA attains accuracy comparable to DivPrune with fewer FLOPs and similar KV storage. On Qwen3-VL, it improves accuracy over both reducers, using fewer FLOPs than training-based TwigVLM, with comparable KV storage.

Table R8. Full-pipeline FLOPs and post-prefill KV-cache comparison on TextVQA at Table 1 operating points. FLOPs include vision encoding, grounding/reduction, auxiliary modules, and LLM inference. Post-prefill KV storage includes auxiliary caches.

BackboneMethodAccuracy (%) ↑Full-pipeline FLOPs (T) ↓KV cache (MiB) ↓
LLaVA-NeXT-7BDivPrune49.64.2881.34
LLaVA-NeXT-7BSparseVLM51.04.90134.39
LLaVA-NeXT-7BRUTA49.73.7381.55
Qwen3-VL-8BDivPrune62.110.5320.04
Qwen3-VL-8BTwigVLM62.612.3320.04
Qwen3-VL-8BRUTA73.112.0119.94

 

Practical system efficiency. Table R8 shows advanced full-pipeline trade-offs including grounding: RUTA attains accuracy comparable to DivPrune with fewer FLOPs on LLaVA-NeXT, and outperforms TwigVLM in accuracy, FLOPs, and KV storage on Qwen3-VL. Relative to DivPrune on Qwen3-VL, it gains accuracy with more computation and similar KV usage. Appendix A.2 reports a 2.62× end-to-end speedup with lower peak memory on LLaVA-NeXT, and an 8.6× prefill speedup on Qwen3-VL. As discussed there, Qwen’s modest TextVQA context provides limited cache savings to offset added modules, while encoding and localization constrain overall acceleration. Fig. 3 further shows that RUTA approaches the unreduced model’s mean accuracy at larger budgets.

Longer contexts offer greater potential memory savings, consistent with Spatial-TTT’s Qwen3-VL measurements [C]. If RUTA’s approximately 95% compression were maintained at 1M visual tokens, Qwen3-VL-8B’s configuration implies a theoretical BF16 visual-token KV reduction from 137.33 to 6.87 GiB, while the localizer’s parameter memory remains constant. This highlights greater potential for system-level memory savings than in the short-context TextVQA setting.


[A] Jiawei Guo, Feifei Zhai, Pu Jian, Qianrun Wei, and Yu Zhou. (2025). CROP: Contextual Region-Oriented Visual Token Pruning. In Conference on Empirical Methods in Natural Language Processing.

[B] Zeng, Q. S., Li, Y., Wang, Q., Jiang, P. T., Wu, Z., Cheng, M. M., & Hou, Q. (2026). A glimpse to compress: Dynamic visual token pruning for large vision-language models. IEEE Transactions on Circuits and Systems for Video Technology.

[C] Fangfu Liu, Diankun Wu, Jiawei Chi, Yimo Cai, Yi-Hsin Hung, Xumin Yu, Hao Li, Han Hu, Yongming Rao, and Yueqi Duan. 2026. Spatial-TTT: Streaming Visual-based Spatial Intelligence with Test-Time Training. arXiv preprint arXiv:2603.12255.

Dear Reviewer qwhp,

Thank you again for your thoughtful feedback. We have posted point-by-point responses to your comments, including additional experiments and clarifications. As the November 18 discussion deadline approaches, we would be grateful if you could take a look at our rebuttal and let us know whether any concerns remain. We would be happy to provide further clarification.

Thank you once again for your time and consideration.

Best regards,

The Authors of Paper #13392

Reviewer_dMTyOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
1
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This work proposes an input-dependent efficient visual tokenization strategy for vision-language reasoning with large vision-language models (LVLMs). Starting with a statement of the desirable property of adaptive and input-dependent visual token needs, the authors propose a two-step objective where they jointly optimize the standard task loss along with a compression term to ensure that the visual tokens produce compressed representations for improved efficiency. The authors attempt to demonstrate the efficacy of the proposed approach on 55 VQA benchmarks.

Critical strengths

The work has the following two primary strengths:

  • Interesting Re-Formulation of Existing Methods

    • The work's two-step objective formulation, although following from existing works, is interesting and does not exist in the prior art to the best of my knowledge. Although the narrative of this section was not easy to follow with many ad-hoc definitions, it presents a non-existent way to frame an existing problem established by existing methods.
  • Decent Ablations

    • The work has a decent number of ablations on the grounding model choice (G. DINO vs OWL etc.) and the loss weighting term, λ\lambda.
Critical weaknesses

The work has the following primary weaknesses:

  • W1: Novelty Concerns
    • As it stands, the work is over-claiming its novelty as it omits citing many relevant works [A, B, C, D], all of which more or less utilize a strategy to predict the number of active visual tokens in an input-dependent fashion, and fails to compare against two that it's actively citing [E, F]. Ideally, the work should at least cite all of these works, clarify precisely how it differs from them and quantitatively compare against them. If there is a reason for omitting these works, the authors should precisely clarify and thoroughly discuss it in their work.

  • W2: Concerns with Added Computational Costs and Unfair Comparisons
    • RUTA leverages a whole new grounding LVLM, be it G. DINO or another, compared to the baselines, which renders the existing comparisons, e.g. at Table 1, not really apples-to-apples. Accordingly, comparing the baselines and RUTA purely based on how many visual tokens they output is not a fair comparison. The authors should instead compare their proposed methods with the baselines on equal grounds, e.g. inference FLOPs, KV Caches and/or other established efficiency measures.
    • Importantly, Table 2 in the Appendix highlights another cost issue with RUTA: Although providing a good improvement with the LLaVA-NeXT backbone, RUTA seems to provide an undesirable trade-off for Qwen3, bringing in a major performance depreciation at matched E2E inference, peak memory and comparable throughput, which does not reflect the difference the authors state w.r.t. the number of visual tokens.

  • W3: Problems with Experiments and Their Presentation
    • The work only reports results on 55 VQA benchmarks, which is very limited for the capabilities of LVLMs used in the work. However, based on the backbones utilized by the authors, and also considering the contemporary works in the literature [A, G, H]. Notably among these works, [G, H] note that many existing generic VQA benchmarks are not sensitive to visual token reduction/compression in the first place. Accordingly, the proposed method should be compared against the baselines under more benchmarks, including but not limited to ChartQA, DocVQA, CV-Bench and RefCOCO.
    • Finally, although the authors note that video applications are a crucial testing bed for visual token compression in their abstract, there were no video benchmarks in which RUTA was evaluated under.

  • W4: Pipeline Complexity and Associated Issues
    • As described in Section 3, the pipeline is relatively complicated compared to the existing works in the literature, while introducing many new hyperparameters for thresholding, temperature scaling and post-hoc processing.
    • On this end, and also for ablations, two things are not very clear: (i) how sensitive is the performance of the proposed method w.r.t. each of these new hyperparameters, and (ii) under which experimental setting hyperparameter tuning was performed. If it is on the TextVQA (assuming the ablation's set-up), the authors should clarify precisely which sets were used for tuning and which were used for reporting the results and why was this benchmark chosen in the first place.
    • That being said, I acknowledge the other ablations presented by the authors, as I noted as a strength in my review.

[A] Huang, W., Zhai, Z., Shen, Y., Cao, S., Zhao, F., Xu, X., ... & Lin, S. (2025, May). Dynamic-llava: Efficient multimodal large language models via dynamic vision-language context sparsification. In International Conference on Learning Representations (Vol. 2025, pp. 69927-69955).

[B] Takezoe, R., Li, Y., Bo, Z. H., Hou, A., Guang, M., & Long, K. (2026, April). Learnpruner: Rethinking attention-based token pruning in vision language models. In International Conference on Learning Representations (Vol. 2026, pp. 66381-66400).

[C] Zhu, J., Zhu, Y., Lu, X., Yan, W., Li, D., Liu, K., ... & Zha, Z. J. (2025). VisionSelector: End-to-end learnable visual token compression for efficient multimodal LLMs. arXiv preprint arXiv:2510.16598.

[D] Wang, Z., Chen, J., Zhou, W., Zhu, H., Liang, J., Shan, L., ... & Qin, B. (2024, May). Smarttrim: Adaptive tokens and attention pruning for efficient vision-language models. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024) (pp. 14937-14953).

[E] Ye, X., Gan, Y., Ge, Y., Zhang, X. P., & Tang, Y. (2025, June). Atp-llava: Adaptive token pruning for large vision language models. In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 24972-24982). IEEE.

[F] Zeng, Q. S., Li, Y., Wang, Q., Jiang, P. T., Wu, Z., Cheng, M. M., & Hou, Q. (2026). A glimpse to compress: Dynamic visual token pruning for large vision-language models. IEEE Transactions on Circuits and Systems for Video Technology.

[G] Endo, M., Wang, X., & Yeung-Levy, S. (2025, October). Feather the throttle: Revisiting visual token pruning for vision-language model acceleration. In 2025 IEEE/CVF International Conference on Computer Vision (ICCV) (pp. 22826-22835). IEEE.

[H] Kuzucu, S., Tonioni, A., Lup, V., Schiele, B., Tombari, F., & Naeem, M. F. (2026). PARCEL: Pool-Anchored Resampling with Conditioned Elastic Queries for Efficient Vision-Language Understanding. arXiv preprint arXiv:2605.30126.

Questions for authors
  • Q1: Could the authors clarify how their work compares against the works mentioned under W1: Novelty Concerns above, cite and at least qualitatively compare against existing similar works?

  • Q2: Could the authors precisely mention how does RUTA compares against the baselines with established efficiency measures, such as inference FLOPs and KV Cache usage?

  • Q3: Could the authors comment on the undesirable trade-off brought in by RUTA w.r.t. Qwen3? What is the opinion of the authors on the generalizability of their approach, given the Qwen3 numbers and the complexity of their pipelines?

  • Q4: Following Q3 above, could the authors report precisely under which regime they performed hyperparameter tuning and how well does the proposed method work across different hyperparameter combinations?

  • Q5: Could the authors report more results with more benchmarks, including but not limited to those mentioned under W3: Problems with Experiments and Their Presentation?


That being said, I am open to discussion by authors and would like to encourage them to address my concerns regardless of my initial score.

9 comments

Thank you for highlighting the relevant prior work and the need for broader evaluation. We clarify our contribution and extend the comparisons as follows.


W1 & Q1: Core contribution, prior work, and comparison coverage.

Core contribution: From an information-theoretic perspective, RUTA treats visual token reduction as task-oriented lossy compression. Under a fixed backbone and numerical precision, expected token count serves as a model-relative rate, while downstream task loss measures distortion. Eqs. (2), (9), and (14) optimize ℓtask+λrateR^ϕ\ell_{\mathrm{task}}+\lambda_{\mathrm{rate}}\widehat R_\phi, encouraging capacity allocation according to the task value of visual evidence under a shared rate penalty.

 

Prior work. Adaptive pruning and task–cost optimization have important precedents (Table R3). RUTA jointly optimizes the evidence available for compression and its per-input representation rate before the LLM. Query-conditioned localization concentrates resolution on relevant evidence. Learned allocation assigns capacity to each image–query pair, while merging preserves non-anchor context within that capacity. Our ablations support this distinction: input-specific allocation outperforms shuffled budgets at similar mean rates, and merging improves over anchors alone (Figs. 4(c), 6(b)).

Table R3. Comparison with relevant prior work in token-count control and evidence processing.

WorkToken-count controlEvidence processing
A: Dynamic-LLaVAPrescribed visual retention ratio at inferenceLearned context sparsification inside the LLM
B: LearnPrunerPrescribed stage budgets at inferenceLearned post-encoder selection and mid-LLM attention-based pruning
C: VisionSelectorSupplied top-K budgetDifferentiable token selection before the LLM
D: SmartTrimLearned instance-dependent masksToken/head pruning within unimodal and cross-modal encoders
E: ATP-LLaVALearned instance- and layer-dependent thresholdsImportance-based and spatial pruning inside the LLM
F: GlimpsePrunePredicted input-dependent masksDecoder glimpse attention and visual features guide pruning; additional region supervision
G: FEATHERPrescribed stage retention ratiosTwo-stage in-LLM pruning using RoPE-free attention and spatial sampling
H: PARCELExternally specified elastic budgetPooled anchors and pool-conditioned query resampling before the LLM
RUTALearned per-image–query count under a rate penaltyQuery-conditioned ROI re-encoding and non-anchor merging before the LLM

 

Comparison coverage. As described in Section 4.2, the original comparison covers attention/query relevance, similarity, diversity, and trainable efficiency methods, including PruneSID’s input-dependent compression rates. As suggested, we add open source methods from Table R3, i.e. Dynamic-LLaVA, VisionSelector, GlimpsePrune, and FEATHER, in Table R4, using the official implementations we could access, adapted to Qwen3-VL-8B under the same evaluation protocol. We will cite all eight suggested works and clarify these distinctions in the revision.

Table R4. Additional comparisons on Qwen3-VL-8B/TextVQA. Token counts are averaged over samples and all main-decoder layers. † GlimpsePrune additionally uses publicly released region annotations on its training subset.

MethodAccuracy (%) ↑Mean visual tokensFull-pipeline FLOPs, T ↓Prefill KV, MiB ↓
Unreduced reference82.31,33729.32198.28
Dynamic-LLaVA52.475.310.6720.78
VisionSelector65.070.010.5720.04
GlimpsePrune†64.475.310.7610.33
FEATHER48.671.810.6520.31
RUTA73.170.412.0119.94

RUTA achieves the highest accuracy among these reducers at comparable token budgets, with additional computation. GlimpsePrune achieves the smallest cache. This supports RUTA’s utility advantage while accounting for its grounding cost. Meanwhile, deployment optimization offers further scope to reduce grounding latency of RUTA: published benchmarks report 4.5× higher throughput for Grounding DINO-T with TensorRT on A100 (9.4→42.6 FPS; [I], Table 5).


[I] Tianhe Ren, Qing Jiang, Shilong Liu, Zhaoyang Zeng, Wenlong Liu, Han Gao, Hongjie Huang, Zhengyu Ma, Xiaoke Jiang, Yihao Chen, Yuda Xiong, Hao Zhang, Feng Li, Peijun Tang, Kent Yu, and Lei Zhang. (2024). Grounding DINO 1.5: Advance the “Edge” of Open-Set Object Detection]. arXiv preprint arXiv:2405.10300.

W2 & Q2: Full-pipeline computational cost and comparison fairness.

Table R4 reports full-pipeline FLOP estimates and measured prefill KV-cache storage, alongside accuracy and token count. FLOPs include vision encoding, online grounding, token reduction, auxiliary modules, and LLM processing—including computation before pruning and answer decoding—with one multiply–add counted as two FLOPs. KV is the actual surviving K/V storage after prefill, with shared storage counted once.

We further clarify our rationale for using token count as an efficiency metric. Under a fixed backbone and numerical precision, tokens provide an actionable representation budget: they can be selected, merged, and routed according to task utility. Aggregate FLOPs and KV storage measure the resulting cost without identifying which evidence receives the budget. We therefore optimize token rate jointly with utility and report full costs to account for method-specific overhead.


W2 & Q3: Qwen3-VL’s efficiency trade-off and generalizability.

On Qwen3-VL, RUTA’s rate–utility curve remains above the compared reducers across the reported budgets and approaches the unreduced model’s mean accuracy at the largest budget (Fig. 3). Table 2 also reports 16.5× and 8.6× prefill speedups on LLaVA-NeXT and Qwen3-VL, respectively. As discussed in Appendix A.2, TextVQA’s modest visual context provides limited absolute KV savings to offset the additional modules’ memory, while encoding and localization costs constrain the overall acceleration.

These absolute KV savings grow with visual context, consistent with the Qwen3-V scaling results in Spatial-TTT [J]. Extrapolating RUTA’s observed approximately 95% compression to a million-token visual sequence would reduce its theoretical BF16 KV storage from 137.33 to 6.87 GiB—a 130.46 GiB saving—while the localizer’s parameter memory remains constant.

Thus, the differing end-to-end improvements are consistent with workload-dependent savings across backbones. Our measured improvements actually establish effectiveness on both evaluated VLMs. Together with the rate–utility results, these gains support cross-backbone applicability.


W4 & Q4: Hyperparameter tuning protocol and sensitivity.

We select all hyperparameters through K-fold cross-validation within the TextVQA training split. TextVQA is chosen because scene-text recognition tests preservation of fine visual evidence. We evaluate the same candidate configurations on both backbones and selected a shared non-rate setting based on fold-averaged accuracy–token trade-offs, calibrating only λrate\lambda_{\mathrm{rate}} separately.

Each backbone’s λrate\lambda_{\mathrm{rate}} setting remains unchanged across configurations. Across all evaluated configurations, the maximum–minimum accuracy spread is 0.6 percentage points on LLaVA-NeXT and 0.3 on Qwen3-VL. Table R5 reports representative configurations with accuracy and token count, ordered as (δ,τ,σsa,σsp2,λsp,β)(\delta,\tau,\sigma_{\mathrm{sa}},\sigma_{\mathrm{sp}}^2,\lambda_{\mathrm{sp}},\beta). Accuracy and token usage remain stable under these joint perturbations on both backbones, supporting the shared configuration within the tested ranges.

Table R5. Hyperparameter sensitivity on TextVQA with LLaVA-NeXT and Qwen3-VL.

ConfigurationHyperparametersLLaVA-NeXT Acc. (%)LLaVA-NeXT TokensQwen3-VL Acc. (%)Qwen3-VL Tokens
Default(0.5, 0.7, 0.07, 0.04, 1, 4)49.750.473.170.4
1(0.4, 0.5, 0.07, 0.04, 1, 4)49.951.673.072.1
2(0.6, 1.0, 0.07, 0.04, 1, 4)49.549.272.869.2
3(0.5, 0.7, 0.05, 0.08, 1, 2)49.650.872.971.0
4(0.5, 0.7, 0.10, 0.02, 1, 6)49.350.073.169.7

[J] Fangfu Liu, Diankun Wu, Jiawei Chi, Yimo Cai, Yi-Hsin Hung, Xumin Yu, Hao Li, Han Hu, Yongming Rao, and Yueqi Duan. 2026. Spatial-TTT: Streaming Visual-based Spatial Intelligence with Test-Time Training. arXiv preprint arXiv:2603.12255.

W3 & Q5: Additional benchmarks and scope

Thanks for your suggestion. Given the available rebuttal resources, we prioritize the four suggested benchmarks, i.e. ChartQA, DocVQA, CV-Bench-2D, and RefCOCO, to directly test whether compression preserves fine-grained visual evidence. We compare RUTA with original and newly added baselines across chart/document understanding, counting and spatial relations, and precise localization. Since CV-Bench-2D has no training split, trainable methods use TextVQA-trained checkpoints.

Table R6. Additional benchmark results with Qwen3-VL-8B. Each entry reports [mean visual tokens] / [score]. ‡ GlimpsePrune is omitted on ChartQA as the released region-supervised subset does not cover this dataset.

MethodChartQADocVQACV-Bench-2DRefCOCO testARefCOCO testB
Unreduced reference1353.5 / 85.51256.7 / 94.81331.3 / 80.91340.5 / 89.01339.8 / 83.0
DivPrune119.0 / 44.4190.0 / 53.284.0 / 68.968.0 / 36.468.0 / 32.4
TwigVLM119.2 / 62.1190.0 / 55.685.0 / 60.568.3 / 47.368.3 / 47.4
Dynamic-LLaVA119.2 / 77.7189.5 / 83.084.3 / 61.568.2 / 74.968.2 / 70.4
VisionSelector119.0 / 56.4190.0 / 77.384.0 / 66.068.0 / 47.068.0 / 53.6
GlimpsePrune—‡189.5 / 50.384.3 / 61.168.2 / 50.768.2 / 48.2
FEATHER118.6 / 50.5191.0 / 51.788.7 / 59.668.5 / 37.368.3 / 36.5
RUTA119.1 / 64.4189.5 / 61.084.3 / 69.765.2 / 75.069.0 / 71.8

At these severe budgets, RUTA leads among reducers on CV-Bench-2D and RefCOCO. Its weaker ChartQA/DocVQA results may reflect a mismatch between object grounding and queries requiring linked text, chart elements, and document layout, which is induced by the localizer. The localizer ablation (Fig. 4(b)) demonstrates that RUTA accommodates different localizers, providing a route to text- and layout-aware localization while retaining the rate–utility framework.


Thank you for your willingness to discuss these concerns. We welcome any further questions.

Reviewer_8r3rOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

RUTA is a trainable pre-LLM visual-token reduction method that learns both which visual evidence to retain and how many tokens to allocate to each image–query pair. It grounds phrases extracted from the query and encodes the smallest bounding box enclosing the selected regions, falling back to the full image when no region passes the confidence threshold. A retention network then predicts Bernoulli probabilities whose sum serves as a differentiable estimate of the token rate. At inference time, this estimate determines an adaptive top-K, and information from non-retained tokens is merged into the retained anchors using semantic and spatial affinities.

The method is evaluated on five single-image VQA benchmarks with LLaVA-NeXT-7B and Qwen3-VL-8B. At the approximately matched low-rate operating points reported in the paper, it obtains the highest mean accuracy among the evaluated reducers.

Critical strengths

I like the paper’s focus on allocating a limited visual-token budget according to the current query rather than applying one fixed budget to every example. In the main results, RUTA achieves the highest mean accuracy among the evaluated reducers on both backbones. The rate–utility curves further suggest that its advantage is most visible when the token budget is tight.

The component studies are also helpful. On TextVQA with Qwen3-VL, the fixed-rate, shuffled-rate, and adaptive-rate comparison suggests that varying the budget is not enough by itself: the budget must also be assigned to the appropriate input. The anchor-merging study indicates that semantic routing provides most of the gain and that spatial routing adds a smaller complementary benefit.

Critical weaknesses

My main concern is the comparison protocol. RUTA receives downstream task supervision and updates several additional modules, whereas most methods in the main table are training-free. The paper gives too little information about how the training-based baselines were trained and tuned, and it omits relevant localization and adaptive-count methods. This makes the size of RUTA’s method-level advantage difficult to judge.

The system-level efficiency evidence is also mixed. On Qwen3-VL, the large token reduction produces only a modest end-to-end speedup and slightly higher peak memory. On LLaVA-NeXT, part of the gain comes from replacing multi-view input with a single crop. A comparison against reduced baselines at matched accuracy would be more informative. The paper should also report variability and explain why the TextVQA results differ across its main table, ablation, and efficiency appendix.

Questions for authors

How were the training-based baselines trained and tuned, and can relevant localization and adaptive-count methods be compared under aligned supervision?

How are the Grounding DINO grounding heads optimized through thresholding, box union, and cropping?

Can the authors report variability, reconcile the differing TextVQA results, and compare full-pipeline efficiency with reduced baselines at matched utility?

5 comments

We thank the reviewer for the constructive feedback and for recognizing the value of query-dependent allocation and our component studies. We address the concerns below.


W1 & Q1: Baseline training and comparison with relevant localization and adaptive-count methods

Training-free methods provide practical references for token reduction without task-specific training. To assess whether RUTA’s advantage persists when competing methods also receive downstream supervision, we include FastVLM and TwigVLM, trained on the corresponding benchmark’s training split using the same answer labels as RUTA. We follow their public training recipes with adaptations to the target backbones and token budgets.

  • FastVLM: We initialize FastViTHD from the authors’ released 7B Stage-2 checkpoint and connect it to the corresponding LLaVA-NeXT or Qwen language backbone. We perform projector alignment followed by joint encoder–projector fine-tuning, keeping the language backbone frozen. Inference resolution is adjusted to approach the target token budget.
  • TwigVLM: We train the three-layer Twig branch and its output normalization/head using answer supervision, while freezing the original vision encoder, multimodal projector, and main LLM. Its retained-token setting is calibrated using the average across all main-decoder layers, including the unpruned prefix and layers after FinalWipe.

RUTA’s higher mean accuracy in Table 1 also holds against these task-trained baselines on both backbones. Thus, the reported advantage extends beyond comparisons with training-free methods. RUTA only uses existing task answers. Its rate penalty relies on internally computed expected token counts and requires no additional downstream region annotations or annotated token budgets.

To further address the missing comparisons with closely related trainable methods, we evaluate Qwen3-VL-8B adaptations of CROP-PLC [A] for localization-based compression and GlimpsePrune [B] for adaptive token allocation on TextVQA. Token counts are averaged over samples and all main-decoder layers, including layers preceding pruning. As shown in Table R1, RUTA still preserves more task utility than both closely related reducers at comparable low token budgets.

Table R1. Comparison with localization-based and adaptive-count methods on Qwen3-VL-8B/TextVQA. † GlimpsePrune retains its additional Visual-CoT region supervision. CROP-PLC uses Qwen2-VL-2B-Instruct for localization, following the original design.

MethodAccuracy (%)Mean visual tokens
Unreduced reference82.31,337
CROP-PLC — localization65.968.0
GlimpsePrune — adaptive-count†64.475.3
RUTA73.170.4

[A] Jiawei Guo, Feifei Zhai, Pu Jian, Qianrun Wei, and Yu Zhou. (2025). CROP: Contextual Region-Oriented Visual Token Pruning. In Conference on Empirical Methods in Natural Language Processing.

[B] Zeng, Q. S., Li, Y., Wang, Q., Jiang, P. T., Wu, Z., Cheng, M. M., & Hou, Q. (2026). A glimpse to compress: Dynamic visual token pruning for large vision-language models. IEEE Transactions on Circuits and Systems for Video Technology.


Q2: Gradient Propagation to the Localizer

The localizer receives gradients through selected box coordinates and differentiable image resampling. Discrete box selection does not detach the selected coordinates from the computation graph:

  • Thresholding (Eq. 5): For the selected set Jδ={j:cj≥δ}J_\delta=\{j:c_j\ge\delta\}, indexing preserves the computation graph of the selected box coordinates. With JδJ_\delta held fixed during backpropagation, gradients flow through these coordinates to the localizer’s box-prediction parameters.
  • Box union (Eq. 5): For selected boxes bj=(xj−,yj−,xj+,yj+)b_j=(x_j^-,y_j^-,x_j^+,y_j^+), we compute broi=(min⁡jxj−, min⁡jyj−, max⁡jxj+, max⁡jyj+)b_{\mathrm{roi}}=\left(\min_j x_j^-,\ \min_j y_j^-,\ \max_j x_j^+,\ \max_j y_j^+\right). These tensor min/max operations are piecewise differentiable, passing coordinate gradients to the boxes that define the enclosing boundaries.
  • Cropping and resizing (Eq. 6): We jointly implement these operations using bilinear grid_sample, with a sampling grid determined by broib_{\mathrm{roi}}. The frozen vision encoder retains gradients with respect to its input.

The objective in Eq. (14) therefore backpropagates along

\longrightarrow Z \longrightarrow \text{resampled image} \longrightarrow b_{\mathrm{roi}} \longrightarrow \\{b_j\\}_{j\in J_\delta} \longrightarrow \varphi$$ This enables the downstream objective to optimize localization through continuous box geometry **conditional on discrete selection**, without requiring auxiliary detection supervision. We will make this gradient path and the differentiable crop implementation explicit in the revision.

W2 & Q3: Efficiency gains, comparisons with reduced baselines, and result consistency.

Clarification of system-level efficiency:

  • Qwen3-VL. As explained in Appendix A.2, RUTA reduces LLM-prefill latency from 363.2 to 42.0 ms (8.6×). Approximately 1,300 visual tokens offer limited absolute KV-cache savings, which can be outweighed by the added localization memory. Longer visual contexts offer substantially larger potential savings: using Qwen3-VL’s official configuration, a million visual tokens require approximately 137.33 GiB of BF16 KV storage. If RUTA’s observed approximately 95% compression were maintained at that length, this would decrease to 6.87 GiB, a theoretical saving of 130.46 GiB, while the localizer’s parameter memory remains unchanged. This identifies a regime where RUTA could deliver much greater benefits on Qwen3-VL than Table 2(b) reveals.
  • LLaVA-NeXT. Reducing input length through query-conditioned cropping is precisely an intended benefit of RUTA’s candidate construction. LLaVA-NeXT’s AnyRes multi-view expansion magnifies this benefit: RUTA reduces both vision-encoding workload and the visual sequence entering the LLM. The reported end-to-end measurements include localization overhead.

Comparison at matched accuracy. We compare RUTA with DivPrune and TwigVLM on Qwen3-VL/TextVQA, selecting their operating points to match RUTA’s accuracy, alongside the unreduced reference. RUTA requires the smallest decoder token budget at comparable accuracy, which supports competitive accuracy–cost trade-offs (Table R2).

Table R2. Accuracy and visual-token counts on Qwen3-VL-8B/TextVQA.

MethodAccuracy (%)#Visual Tokens
Unreduced reference82.31337
DivPrune — training-free73.6210.0
TwigVLM — training-based73.3140.0
RUTA73.170.4

Result consistency and variability. Table 1 and Table 2, and the ablation group in Figs. 4 & 6 use separate training and evaluation runs. At similar average token counts, checkpoint differences and inference with different random seeds can change retained evidence and answers, resulting in numerical fluctuations. A supplementary three-seed check yields 71.7±0.25% accuracy and 68.36±5.23 visual tokens on TextVQA. We will harmonize scoring from main results, ablation studies and appendix, and update the affected tables, figures to remove misunderstanding.

Dear Reviewer 8r3r,

Thank you again for your thoughtful feedback. We have posted point-by-point responses to your comments, including additional experiments and clarifications. As the November 18 discussion deadline approaches, we would be grateful if you could take a look at our rebuttal and let us know whether any concerns remain. We would be happy to provide further clarification.

Thank you once again for your time and consideration.

Best regards,

The Authors of Paper #13392

Comments on the paper

1 comment

Dear Reviewers,

The authors have posted their rebuttals and updated their manuscript/responses. Please carefully read the authors' responses and engage in the discussion to clarify any remaining concerns.

In light of the rebuttal, please let the authors know whether your concerns have been adequately addressed, and update your review comments and scores if appropriate. Active participation is essential for us to reach a well-calibrated decision.

Please note that the author-reviewer discussion phase ends on November 18 at 11:59 PM AoE, after which no further responses can be posted. Thank you for your continued time and effort.

Best regards,

Area Chair

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.