acceptodds
Withdrawn from ICLR 2027

Now You See It, Now You Don’t: Rethinking visual token allocation across Vision Language Model layers

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

56 views

Abstract

Vision-language models (VLMs) can often localize relevant regions in an image, yet still fail to perceive the fine-grained visual evidence needed to answer correctly, leading to visual hallucinations. Using more tokens provides a finer representation of the image, but also increases the computation, memory, and latency required for inference. VLMs therefore operate under a limited visual-token budget: spending more of it on irrelevant details wastes computation without improving the answer. The central challenge is therefore to spend this limited budget where additional visual detail is most useful. A common strategy is attention-guided cropping, however, existing approaches typically extract this attention from a manually selected layer or a fixed range of layers, without establishing which depth provides the most reliable signal for visual selection. We show that the choice of layer is a dominant factor in attention-guided visual selection. By systematically masking text-to-image attention across network depth, we show that the information relevant to the model's prediction is concentrated within an early-to-middle portion of the network. This boundary occurs substantially earlier than the layers used by existing attention-based selection methods. Based on this finding, we introduce two complementary strategies. Depth-Weighted Attention (DWA) combines attention information across the full depth of the model through a closed-form ridge regression, learning layer weights that suppress less informative late-layer contributions and produce more reliable spatial selection. Adaptive Visual Resolution (AVR) uses the identified depth boundary to remove visual tokens that no longer affect the prediction and redirects the saved computation toward higher-resolution encoding, allowing the model to represent important image regions with finer visual detail. Across nine matched-compute comparisons, DWA consistently improves over existing layer-selection strategies, achieving gains of 7.9 and 8.9 percentage points over the strongest baseline and locating answer-relevant evidence in 63.9% of cases compared with 13.6%. Against native dynamic-resolution models operating at 5.5 the visual-token budget, DWA and AVR achieve statistically indistinguishable performance while using only 14-18% of the visual-token computation.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Vision-Language Models, Visual Hallucination, Visual Tokenization, Multimodal Reasoning, Efficient Vision-Language Models, Fine-Grained Visual Understanding

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_JEgvOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This work studies how visual information is utilized across LLM layers in VLMs. Through layer-wise causal masking, the authors identify a depth boundary separating early visual-information transport from later spatial readout. Based on this observation, they propose two methods: Depth-Weighted Attention (DWA), which aggregates attention patterns across decoder layers to localize informative regions for high-resolution re-encoding, and Adaptive Visual Resolution (AVR), which allocates more visual tokens to earlier layers while restricting visual-token access in later layers. The experiments report improved performance on several fine-grained visual reasoning benchmarks under the authors’ proposed token-budget setting.

Critical strengths

None.

Critical weaknesses
  1. The manuscript suffers from weak logical flow and insufficiently polished presentation. For example, the 1st paragraph of the Introduction jumps from a specific example to broad claims about localization and perception without establishing the reasoning clearly. The Abstract is also unusually fragmented in multiple paragraphs, and several methodological details are missing for reliable reproduction.

  2. The practical motivation of the proposed approach is not convincing. Modern Qwen-VL models already support dynamic-resolution image encoding, raising the question of why relevant regions need to be identified through the LLM decoder and then re-encoded at higher resolution. Although the authors argue that this avoids spending visual tokens on answer-irrelevant regions, DWA itself introduces substantial additional overhead: it requires two LLM prefill passes (one for localization and one for answering), extracting layer-wise attention maps prevents the use of FlashAttention, and the ridge-regression localizer should be fitted separately for different VLM families. The paper therefore lacks a convincing end-to-end analysis showing that the proposed token allocation actually provides a favorable accuracy–efficiency trade-off over the native dynamic-resolution mechanism.

  3. The efficiency claim of AVR is overstated and not supported by actual runtime measurements. In the reported implementation, discarded visual tokens are only masked by adding a (-10^4) bias to their attention logits, while the token tensors remain at their original length. Therefore, no actual token reduction or computational saving is realized during inference. Consequently, the Abstract’s claim that AVR uses “only 14–18% of visual-token computation” is misleading without a physical token-removal implementation and corresponding wall-clock runtime measurements.

  4. The method has a serious reproducibility issue. The main text states that the (2L+7)-dimensional feature vector includes rank, neighbourhood, and geometry features and explicitly refers to Appendix B for their definitions. However, Appendix B does not provide the concrete definitions of these additional features. Since these features are directly used by the ridge regressor to determine crop scores, omitting their exact formulation makes the proposed method impossible to reproduce faithfully from the paper alone.

  5. The experimental section lacks direct comparisons with the competing methods discussed in Related Work. The main results primarily compare DWA/AVR against uncropped equal-budget VLM backbones, rather than representative existing approaches. As a result, the experiments demonstrate improvements over the authors’ own baselines, but do not establish whether the proposed methods are actually competitive with existing visual cropping or token-pruning approaches under matched settings.

  6. The claimed 600-token compute budget raises reproducibility concerns across different VLM architectures. The main results treat 600 visual tokens as an equal-compute reference, while the appendix only states that images are bicubically resized toward a target token count and that the closest realized count is retained within a tolerance. However, models such as LLaVA-OV-7B and InternVL3-8B employ fixed-resolution vision encoders, even though their preprocessing pipelines may dynamically partition an image into multiple tiles. Their resulting visual-token counts are therefore constrained by model-specific tiling and tokenization schemes rather than being freely adjustable through image resizing. The paper does not explain how a 600-token budget (or the two approximately 300-token passes used by DWA) is actually realized for these models, nor does it report the realized token counts or tiling configurations. This makes the claimed matched-compute comparison difficult to verify.

Questions for authors

See the weaknesses.

Reviewer_w8nEOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies visual token allocation in vision-language models by systematically masking text-to-image attention across LLM layers. The analysis shows that prediction-related information is concentrated in early-to-middle layers. Based on these findings, depth-weighted attention (DWA) and adaptive visual resolution (AVR) are proposed to improve fine-grained recognition. Experiments across four VLMs show that the proposed modules could be helpful for fine-grained tasks.

Critical strengths

None

Critical weaknesses
  1. The paper provides limited insights into visual token allocation in VLMs. First, the depth boundary finding has already been pointed out by prior work [1, 2]. Second, the spatial readout diagnosis is closely related to visual grounding with attention heads/layers [3,4]. However, this paper did not cite prior work or compare with them.

  2. There is no comparison with existing methods. The authors did not compare with closely related methods (e.g., ViCrop, ZoomEye, and VisLens), making it difficult to justify the effectiveness of the proposed modules.

Reference

[1] What’s in the Image? A Deep-Dive into the Vision of Vision Language Models. CVPR 2025.

[2] Cross-modal Information Flow in Multimodal Large Language Models. CVPR 2025.

[3] Your Large Vision-Language Model Only Needs A Few Attention Heads For Visual Grounding. CVPR 2025.

[4] Seeing but Not Believing: Probing the Disconnect Between Visual Attention and Answer Correctness in VLMs. ICLR 2025.

Questions for authors

The effectiveness of the proposed modules requires further justification. As shown in Table 2, DWA significantly degrades the accuracy of the baseline on CV-bench and RealWorldQA, raising concerns regarding its generality. Similarly, Table 4 shows that AVR does not consistently improve Qwen2-VL-7B and LLaVA-OV-7B.

Reviewer_pbeyOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates how visual-token budgets should be allocated across image regions and VLM decoder layers. Attention-masking interventions and layer-wise crop-selection experiments suggest that the depths where direct visual access influences predictions differ from those providing useful spatial readouts. Motivated by this observation, Depth-Weighted Attention (DWA) learns a supervised regression over attention-derived features to select a high-resolution crop. Adaptive Visual Resolution (AVR) increases full-image resolution while restricting access to most visual tokens in later layers. Experiments across multiple VLMs and four main benchmarks show improvements on localized-detail tasks and substantial limitations on tasks requiring broader context. AVR preserves much of the higher-resolution model’s accuracy under a nominal visual token-layer budget, although actual computational savings are not demonstrated.

Critical strengths

S1. A useful investigation of attention-readout depth. The paper varies readout depth while controlling the crop and answering pipeline, complemented by single-layer, prefix, and suffix masking experiments (Sec. 4.1; Table 1; Appendix B.1). The substantial variation in crop-selection accuracy across layers convincingly demonstrates that readout depth is an important design choice. The distinction between sensitivity to direct visual access and useful spatial readout provides a valuable empirical insight.

S2. Meaningful local-detail improvements with informative failure cases. DWA improves V*Bench accuracy from 62.3% to 72.8% on Qwen3-VL-2B and from 59.2% to 68.1% on Qwen2-VL-7B under the stated token accounting, with additional improvements on HR-Bench 4K (Table 2). The paper also reports substantial degradation on CV-Bench and RealWorldQA. These negative results help establish when spatial concentration is beneficial and when preserving broader context is necessary.

Critical weaknesses

W1. The incremental contribution is insufficiently separated from added supervision and design choices. DWA fits a bounding-box-supervised regressor using V*Bench annotations, combines attention with rank, neighborhood, and geometry features, and introduces fixed quarter-image crops and outer-grid exclusion. These additions make the pipeline more elaborate than simply improving layer aggregation. The ablations in Appendix B.2 do not isolate the contributions of the auxiliary features and crop heuristics. Consequently, it remains unclear how much improvement reflects a transferable insight about decoder depth versus the additional supervised localization machinery, and whether the benefits justify these choices.

W2. Presentation and reporting issues hinder interpretation and verification. The abstract contains a visible formatting error at L38, while the introduction moves quickly from an illustrative example to contribution claims without sufficiently developing the research gap (Sec. 1). More consequentially, Figure 1 does not clearly distinguish supervised fitting from inference, and Figure 2 suggests actual token removal despite the masking-only implementation described in Sec. 3.5.

Questions for authors

See weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.