- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes RUTA, a learning-based pre-LLM visual token reduction framework designed to jointly determine which visual tokens to retain and how many tokens to allocate for each image-query pair. Specifically, RUTA first identifies query-relevant regions via a query-conditioned localizer, which are then cropped, resized, and re-encoded into candidate tokens. Next, it predicts a retention probability for each candidate token, using the sum of these probabilities as a differentiable estimate of the expected token count. During training, token selection is modeled via independent Bernoulli gates coupled with a straight-through estimator (STE), whereas at inference time, it switches to a deterministic adaptive top-K selection determined by the expected token rate. Non-retained candidate tokens are aggregated into the retained anchor tokens based on semantic affinity and spatial proximity. The entire pipeline is jointly optimized using a downstream task loss and a penalty on expected token usage. The authors evaluate RUTA on LLaVA-NeXT-7B and Qwen3-VL-8B across five visual question answering (VQA) benchmarks, reporting performance gains under low token budgets and providing ablation studies on rate allocation, localizer choices, and anchor merging strategies.
Critical strengths
S1.Problem Formulation and Motivation: The paper formulates visual token reduction as a rate-utility optimization problem inspired by rate-distortion theory. It uses the sum of token retention probabilities as a differentiable estimate of the expected token count and penalizes this estimate to control the trade-off between task performance and token usage. This formulation enables the model to adaptively allocate tokens according to the visual evidence requirements of different image-query pairs, providing a unified optimization framework for mitigating the over-allocation of tokens to simple queries and the under-allocation of tokens to more demanding ones in fixed-budget methods.
S2.Architectural Synergy and Information Preservation: The paper combines query-conditioned region-of-interest (ROI) localization and re-encoding with anchor-based token merging guided by semantic affinity and spatial proximity. This design concentrates visual resolution on query-relevant regions while softly aggregating information from non-anchor tokens into retained anchors. The reported results demonstrate that RUTA maintains competitive task performance at extremely low token budgets, while the merging ablations provide supporting evidence for the benefits of combining semantic and spatial information during aggregation.
S3.Targeted Ablation Studies: The paper conducts informative ablations using fixed-rate, shuffled-rate, and adaptive-rate controls. By comparing these strategies at approximately the same average token budget, the experiments help distinguish the benefits of varying token allocations from those of assigning budgets appropriately to individual inputs. Furthermore, comparisons among anchor-only retention, uniform merging, spatial routing, semantic routing, and the full RUTA method provide targeted evidence regarding the role of non-anchor information aggregation and the contributions of semantic and spatial affinity.
Critical weaknesses
W1. Ambiguous token budget accounting undermines the credibility of the matched-rate comparisons. Table 1 defines "Tok" as the number of visual tokens averaged over samples and LLM layers (consistent with the layer-wise average formulation in Sec. 3.1). However, the reported numbers for in-LLM baselines contradict this definition. For instance, on VQAv2 with LLaVA-NeXT-7B (L=32 layers, N=2,233 reference tokens), an in-LLM reducer that processes full visual tokens in the first m layers before pruning has a strict mathematical lower bound of mN/L >= 69.8 layer-averaged tokens (and >= 139.6 for FastV's default m=2). Yet, Table 1 reports 42.0 tokens for FastV. This discrepancy suggests that either pre-pruning layers were excluded from the average, or different input preprocessing/resolutions were used across baselines, calling into question whether the comparisons in Table 1 and Fig. 3 are genuinely rate-matched.
W2. Unclear gradient pathways in the claimed joint optimization hinder reproducibility. The paper claims to jointly optimize the candidate constructor (updating the localizer's grounding heads) and the rate-controlled compressor via a single end-to-end objective (Eq. 14). However, two critical gradient pathways are unexplained: (1) In Eq. (10)-(13), the straight-through estimator (STE) gates are only applied to the retained anchors in the final output sequence, while non-anchors (xi_j = 0) are indexed out without gate participation in the merging weights (Eq. 12), leaving it unclear how non-anchor retention probabilities receive task gradients to become anchors. (2) Candidate construction (Eq. 4-6) relies on discrete phrase parsing, confidence thresholding, box union, and image cropping, with no explanation of how gradients from the downstream task or rate loss backpropagate through these non-differentiable operations to update the localizer parameters.
W3. Entangled algorithmic gains and limited end-to-end system efficiency advantages. First, while RUTA is fine-tuned end-to-end (including the VLM projector, merging layers, and an external 232M-parameter localizer), most baselines in Table 1 are training-free. Meanwhile, closely related trainable and adaptive reducers discussed in Sec. 2.2 (e.g., CROP, ATP-LLaVA, GlimpsePrune) are omitted from the empirical comparisons, making it difficult to isolate the incremental value of the proposed rate-utility allocation from the benefits of region cropping and task-specific fine-tuning. Second, the system efficiency evaluation in Appendix Table 2 only compares against the unreduced reference rather than competing reducers. On Qwen3-VL-8B, reducing visual tokens by 94.8% (1,337.5 -> 69.3) yields only a 1.12x end-to-end speedup while increasing peak GPU memory (21.0 GB -> 21.7 GB) and dropping accuracy by 9.9 points (83.0% -> 73.1%), which weakens the practical system-efficiency claim.
Questions for authors
Q1. Clarification on token budget accounting and matching protocol in Table 1: How is "Tok" computed for in-LLM baselines such as FastV and SparseVLM? Does it include the unpruned early LLM layers, and do all baselines share the exact same input resolution and multi-view preprocessing? Please explain how FastV achieves 42.0 tokens on LLaVA-NeXT-7B given the layer-average lower bound of mN/L, and clarify how a consistent accounting metric affects the rate-matched rankings.
Q2. Clarification on gradient propagation for joint training: Could you provide pseudocode or a computation graph clarifying: (a) how task gradients backpropagate to the retention probabilities of non-selected tokens (xi_j = 0) and through the anchor-set normalization in Eq. (12)-(13)? and (b) how the localizer's grounding heads are updated across the discrete thresholding, box union, and cropping operations in Eq. (4)-(6) (e.g., whether auxiliary detection losses or differentiable ROI operators are used)?
5 comments
W3. Entangled algorithmic gains and limited end-to-end system efficiency advantages.
Comparison fairness and related methods. Comparisons with training-free reducers test whether RUTA’s training improves inference-time accuracy–compression trade-offs. To address supervision differences, FastVLM and TwigVLM use the same training splits and answer labels as RUTA. We also compare CROP (localization) [A] and GlimpsePrune (adaptive pruning) [B] on TextVQA with Qwen3-VL-8B, preserving their original objectives. GlimpsePrune additionally uses region supervision, whereas RUTA requires only task answers. As shown in Table R7, RUTA outperforms both closely related reducers at comparable low token budgets, complementing the task-supervised comparisons in Table 1.
Table R7. Comparison with CROP-PLC and GlimpsePrune on Qwen3-VL-8B/TextVQA.
| Method | Accuracy (%) ↑ | All-layer mean visual tokens ↓ |
|---|---|---|
| Reference | 82.3 | 1,337 |
| CROP-PLC | 65.9 | 68.0 |
| GlimpsePrune | 64.4 | 75.3 |
| RUTA | 73.1 | 70.4 |
Contribution of adaptive allocation. Fig. 4(c) fixes the trained model and localization pipeline. At approximately 70 tokens, adaptive allocation improves accuracy by 1.7 points over fixed allocation and 2.0 points over shuffled allocation, which preserves the budget distribution. These controls isolate the benefit of assigning capacity to the appropriate input beyond shared cropping and fine-tuning.
Full-pipeline efficiency. We compare RUTA with the two highest-accuracy competing TextVQA reducers on each backbone at the Table 1 operating points (Table R8). On LLaVA-NeXT, RUTA attains accuracy comparable to DivPrune with fewer FLOPs and similar KV storage. On Qwen3-VL, it improves accuracy over both reducers, using fewer FLOPs than training-based TwigVLM, with comparable KV storage.
Table R8. Full-pipeline FLOPs and post-prefill KV-cache comparison on TextVQA at Table 1 operating points. FLOPs include vision encoding, grounding/reduction, auxiliary modules, and LLM inference. Post-prefill KV storage includes auxiliary caches.
| Backbone | Method | Accuracy (%) ↑ | Full-pipeline FLOPs (T) ↓ | KV cache (MiB) ↓ |
|---|---|---|---|---|
| LLaVA-NeXT-7B | DivPrune | 49.6 | 4.28 | 81.34 |
| LLaVA-NeXT-7B | SparseVLM | 51.0 | 4.90 | 134.39 |
| LLaVA-NeXT-7B | RUTA | 49.7 | 3.73 | 81.55 |
| Qwen3-VL-8B | DivPrune | 62.1 | 10.53 | 20.04 |
| Qwen3-VL-8B | TwigVLM | 62.6 | 12.33 | 20.04 |
| Qwen3-VL-8B | RUTA | 73.1 | 12.01 | 19.94 |
Practical system efficiency. Table R8 shows advanced full-pipeline trade-offs including grounding: RUTA attains accuracy comparable to DivPrune with fewer FLOPs on LLaVA-NeXT, and outperforms TwigVLM in accuracy, FLOPs, and KV storage on Qwen3-VL. Relative to DivPrune on Qwen3-VL, it gains accuracy with more computation and similar KV usage. Appendix A.2 reports a 2.62× end-to-end speedup with lower peak memory on LLaVA-NeXT, and an 8.6× prefill speedup on Qwen3-VL. As discussed there, Qwen’s modest TextVQA context provides limited cache savings to offset added modules, while encoding and localization constrain overall acceleration. Fig. 3 further shows that RUTA approaches the unreduced model’s mean accuracy at larger budgets.
Longer contexts offer greater potential memory savings, consistent with Spatial-TTT’s Qwen3-VL measurements [C]. If RUTA’s approximately 95% compression were maintained at 1M visual tokens, Qwen3-VL-8B’s configuration implies a theoretical BF16 visual-token KV reduction from 137.33 to 6.87 GiB, while the localizer’s parameter memory remains constant. This highlights greater potential for system-level memory savings than in the short-context TextVQA setting.
[A] Jiawei Guo, Feifei Zhai, Pu Jian, Qianrun Wei, and Yu Zhou. (2025). CROP: Contextual Region-Oriented Visual Token Pruning. In Conference on Empirical Methods in Natural Language Processing.
[B] Zeng, Q. S., Li, Y., Wang, Q., Jiang, P. T., Wu, Z., Cheng, M. M., & Hou, Q. (2026). A glimpse to compress: Dynamic visual token pruning for large vision-language models. IEEE Transactions on Circuits and Systems for Video Technology.
[C] Fangfu Liu, Diankun Wu, Jiawei Chi, Yimo Cai, Yi-Hsin Hung, Xumin Yu, Hao Li, Han Hu, Yongming Rao, and Yueqi Duan. 2026. Spatial-TTT: Streaming Visual-based Spatial Intelligence with Test-Time Training. arXiv preprint arXiv:2603.12255.
Dear Reviewer qwhp,
Thank you again for your thoughtful feedback. We have posted point-by-point responses to your comments, including additional experiments and clarifications. As the November 18 discussion deadline approaches, we would be grateful if you could take a look at our rebuttal and let us know whether any concerns remain. We would be happy to provide further clarification.
Thank you once again for your time and consideration.
Best regards,
The Authors of Paper #13392
We thank the reviewer for the careful assessment and for highlighting the token-accounting and gradient-flow questions. We clarify these points and the comparison protocol below.
W1 & Q1: Clarification on token budget accounting and matching protocolThank you for identifying this inconsistency. Tok denotes the number of visual tokens averaged over samples and LLM layers for in-LLM baselines. However, the caption omitted the markers identifying FastV’s Tok as its retained count after two unpruned layers, denoted here by tF, and SparseVLM’s Tok as its average count over the remaining layers, denoted by tS. For both methods, we preserve these unpruned layers to avoid severe accuracy degradation from premature pruning. Most of baselines use the reference model’s input resolution, and multi-view preprocessing within LLaVA-NeXT. However, there are exceptions, such as FastVLM, which modifies the vision encoder and input resolution to improve efficiency.
Specifically, in our SparseVLM experiments, pruning occurs after zero-indexed layers [1, 6, 15]. Let N denote the input visual-token count, L the decoder depth, and rj the count after each pruning stage, including recycled tokens. Then tS=L−25r1+9r2+(L−16)r3. The corrected all-layer token counts, including the two unpruned layers, are $ \mathrm{Tok}_{\mathrm{FastV}}^{\mathrm{corr}} =\frac{2N+(L-2)t_{\mathrm{F}}}{L}, \mathrm{Tok}_{\mathrm{SparseVLM}}^{\mathrm{corr}} =\frac{2N+(L-2)t_{\mathrm{S}}}{L}. $ These quantities are averaged over samples. For example, on VQAv2, FastV’s corrected counts are approximately 178.9 on LLaVA-NeXT and 102.8 on Qwen3-VL; SparseVLM’s are 179.0 and 103.0, respectively. We will add the missing markers in the caption of all affected tables and figures.
RUTA retains the highest mean accuracy on both backbones, while consistent accounting assigns larger token budgets to FastV and SparseVLM. Its advantage at the evaluated operating points therefore remains supported.
W2 & Q2: Clarification on gradient propagation for joint trainingNon-selected tokens and anchor normalization (Eqs. 10–13). The training implementation includes an STE factor omitted from Eq. (13). For simplicity, the expression for Δzi in Eq. (13) omitted the factor 1−ξ~j. The full expression is
\frac{\sum_{j\in\bar{A}}w_{i,j}(1-\tilde{\xi}_j)W_{\mathrm{value}}z_j} {\max\\!\left(1,\sum_{j\in\bar{A}}w_{i,j}\right)}, \qquad i\in A.$$ For non-selected tokens, $1-\tilde{\xi}_j=1$ in the forward pass, preserving the stated aggregation. Its Binary-Concrete backward relaxation provides **task gradients to their retention probabilities** $\rho_j$, alongside the rate gradient from Eq. (9). Gradients also propagate through the routing softmax and aggregation denominator, with the sampled anchor and non-anchor sets held fixed. **Localizer optimization (Eqs. 4–6).** Threshold-based indexing preserves the computation graph of selected box coordinates. Coordinate-wise minima/maxima construct the enclosing box, and bilinear `grid_sample` implements differentiable cropping and resizing. The frozen vision encoder preserves gradients with respect to its input, giving the backward path $$\ell_{\mathrm{task}}+\lambda_{\mathrm{rate}}\widehat R_\phi \\;\longrightarrow\ Z \\;\longrightarrow\ \text{resampled image} \\;\longrightarrow\ b_{\mathrm{roi}} \\;\longrightarrow\ \text{selected box coordinates} \\;\longrightarrow\ \varphi.$$ Thus, the downstream objective updates the localizer through continuous coordinates conditional on the selected boxes, **without auxiliary detection losses or additional downstream box annotations**. We will correct Eq. (10) to specify the Binary-Concrete backward relaxation, restore the omitted factor in Eq. (13), and make the gradient flow during training more explicit in the revision.