acceptodds
Withdrawn from ICLR 2027

Vision-Language Models Do Not Keep Looking

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Autoregressive Large Vision-Language Models solve multimodal reasoning tasks by retaining hundreds or thousands of visual tokens, allowing every generated token to attend to the visual tokens at any time during the generation process. However, whether continuous access to these visual tokens is necessary remains poorly understood. In this work, we investigate how visual token influence evolves along two dimensions: generation time and transformer depth. Using long-form visual reasoning benchmarks and multiple model scales, we first analyze the attention dynamics across generation time and transformer depth. We identify a temporal and layer-wise structure in the use of visual information. Visual attention peaks during the initial generation steps and then rapidly declines as attention shifts toward previously generated reasoning tokens. Across depth, visual influence is concentrated in intermediate layers and becomes very weak in the final layers. Motivated by these observations, we propose a simple intervention that removes all visual tokens across generation time and model depth. These interventions confirm a transition from early multimodal integration, during which visual information is incorporated into the generated token representations, to later generation, where the model relies primarily on these accumulated representations rather than direct access to the visual tokens. Thus, retaining the complete visual context at every layer and decoding step may be unnecessary. Our experiments demonstrate that we can remove up to 59% of visual tokens across generation time and 11% across model depth, while incurring only a negligible performance drop.

Primary area:
foundation or frontier models, including LLMs
Keywords:
vlm, vision-language, mllms, cot, reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_37sMOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates visual-token usage across generation time and transformer depth on multimodal reasoning tasks. Experiments on four Qwen3-VL model sizes show that visual attention concentrates early in generation and in intermediate layers. The authors then remove visual KV-cache entries at selected decoding steps or upper layers, finding that some direct visual access can be removed with limited accuracy degradation. The appendices provide analytical estimates of computation and cache savings.

Critical strengths
  1. Jointly examining generation time and transformer depth provides a useful perspective on visual access during reasoning.
  2. The paper reports task-dependent accuracy-reduction trade-offs and provides an analysis of attention computation and KV-cache savings.
Critical weaknesses
  1. The incremental contribution over prior visual-attention studies is limited. Prior work already investigates layer-dependent visual-token redundancy and motivates reducing visual processing in later layers. The paper acknowledges this literature, but its main distinction is the focus on long-form multimodal reasoning and complete visual-token removal. This is a useful extension, yet the additional insight needs clearer substantiation.
  2. Practical runtime and inference-engine compatibility are not demonstrated. Appendix B provides analytical FLOP and KV-cache estimates, but no measured end-to-end latency, throughput, or peak GPU memory. Logical cache reduction does not automatically translate into equivalent practical savings. The authors should clarify compatibility with optimized engines such as vLLM, particularly for layer-specific cache lengths and mid-generation eviction, and measure speed and memory against the unmodified engine.
  3. The efficiency benefit at near-baseline accuracy appears modest and task-dependent.
  4. Generalization across model architectures remains unverified. All experiments use Qwen3-VL. Scaling within one family does not establish that the temporal integration window or depth-wise pattern generalizes across architectures. An additional family, such as Gemma 4 12B, would strengthen the findings.
  5. Multi-turn dialogue may invalidate the assumption behind permanent visual-cache removal. The experiments examine continued generation for a fixed question. In multi-turn dialogue, a later question may concern a different object or previously unused detail, which representations accumulated for the first response may not preserve. A controlled second-turn evaluation with substantially different visual questions would clarify this limitation.
Questions for authors

How much post-removal robustness comes from image-conditioned prompt-token states versus previously generated reasoning-token states? Can these sources be separated to support the proposed explanation?

Reviewer_peeYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies the trade-off between accuracy and cost when retaining visual tokens during long-form generation. Attention measurements over time and depth motivate removing visual KV entries after a token threshold or in upper layers. Experiments cover four Qwen3-VL sizes and four benchmarks, including combined interventions. The results report accuracy and output length alongside analytical estimates of attention computation and cache savings.

Critical strengths

The simple intervention makes its effects reasonably easy to interpret. Separate sweeps over time and depth show how the effect of removing visual access depends on when and where it happens. Testing several model sizes helps assess whether a policy transfers across capacities. Reporting generation lengths alongside accuracy also helps judge whether the apparent savings would be useful in practice.

Critical weaknesses

The claim of negligible performance loss needs uncertainty estimates for the accuracy differences in the main results. Small changes are difficult to interpret without knowing how much the baseline varies. As a supporting example, the Qwen3-VL-2B HallusionBench baseline is 67.82 in the main table and 69.09 in Appendix A. That difference matches or exceeds several effects described as negligible. Different runs or settings could explain it, but the manuscript needs to identify them and report the corresponding uncertainty.

Section 4 also needs to specify what removing visual KV entries during prefill actually skips. Deleting cached K/V after a layer has run would not save that layer's visual-row projections or MLP work. The cost calculation in Appendix B.2, Equation 12, assumes fewer visual rows are processed in upper layers. The claimed deployment savings depend on whether the implementation performs that operation.

Questions for authors
  1. What uncertainty surrounds the accuracy differences described as negligible? Please report paired estimates and specify the decoding settings, seeds, or other run differences that explain the inconsistent unmodified baselines.

  2. Does the depth intervention in Section 4 remove visual hidden-state rows before upper-layer computation, or only remove their KV entries? Please explain which prefill computations are skipped and reconcile them with the reported cost reduction.

Reviewer_P7TkOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies how Large Vision-Language Models (LVLMs) use visual tokens over the course of long-form multimodal reasoning. It analyzes attention allocation across generation time and Transformer depth, and reports that visual attention generally decreases as generation proceeds and is concentrated more strongly in intermediate layers.

Critical strengths
  1. Understanding when visual information remains necessary during long multimodal generation is relevant both to model-mechanism analysis and to inference efficiency.

  2. The paper examines both generation time and Transformer depth. This two-dimensional analysis is more informative than studying only a single attention layer or a single decoding position.

Critical weaknesses
  1. Decreasing attention allocation does not establish that visual information has been fully integrated. Attention weights are not a direct measure of causal contribution to the final prediction. In addition, as text context grows during generation, the allocation of attention across visual and textual positions can change even if some visual evidence remains important. The paper does not sufficiently distinguish attention allocation from the actual contribution of visual information to the model's representations and outputs.

  2. All evaluated models belong to the Qwen3-VL family. Different LVLMs may use different visual encoders, tokenization schemes, and cross-modal fusion mechanisms, so the findings may not transfer directly to other architectures.

  3. The practical efficiency claims are supported mainly by theoretical FLOPs and KV-cache reduction estimates. The paper does not provide sufficiently comprehensive end-to-end measurements of latency, throughput, and peak GPU memory. Moreover, removing visual KV entries does not eliminate the visual encoder's initial computation, so the overall deployment benefit may be smaller than the attention-only savings suggest.

Questions for authors
  1. How do the authors distinguish decreasing attention allocation from the actual causal contribution of visual information to the generated representations and final predictions?

  2. Can the authors validate the temporal and depth-wise findings on at least one LVLM architecture outside the Qwen3-VL family?

  3. Can the authors report end-to-end latency, throughput, and peak-memory measurements, including the cost of the visual encoder, rather than relying primarily on theoretical attention FLOPs and KV-cache reductions?

Reviewer_MnViOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates how visual token attention evolves across generation time and transformer depth in long-form VLM reasoning. The authors observe that visual attention is concentrated in early generation steps and intermediate transformer layers, suggesting a Visual Integration Window in which visual information is incorporated into generated token representations. Based on these observations, the paper introduces causal interventions that remove visual KV cache entries at different generation steps and transformer layers. Experiments on four multimodal reasoning benchmarks using Qwen3-VL models of different scales show that visual tokens can be removed at certain stages with limited performance degradation, suggesting opportunities to reduce memory and computational costs.

Critical strengths
  1. Interesting Research Question: The paper investigates how visual token dependence evolves across generation time and transformer depth, providing a useful perspective on understanding visual information in VLM reasoning.

  2. Causal Intervention Experiments: Beyond attention analysis, the paper conducts visual KV cache removal experiments to examine the importance of visual tokens at different generation steps and transformer layers, providing empirical support for its observations.

Critical weaknesses
  1. Insufficient Efficiency Evaluation: Although the paper reports absolute FLOPs and memory savings after removing the visual KV cache, it lacks relative reduction percentages and actual inference speedup measurements, making it difficult to assess the practical effectiveness of the proposed method.

  2. Limited Novelty and Mechanistic Analysis: The observed attention patterns across generation time and transformer depth lead to relatively straightforward conclusions about how VLMs utilize visual information, without sufficient in-depth analysis. For example, the proposed Visual Integration Window lacks detailed experimental validation and case studies to explain how visual information is integrated into early generated token representations during reasoning.

Questions for authors
  1. Case Studies and Dataset Analysis: How do the characteristics of different benchmark datasets relate to the observed experimental results? Can additional case studies help explain these differences?

  2. Generalizability: Do similar temporal and layer-wise patterns hold across other VLM model families beyond Qwen3-VL?

  3. Implementation Details: How are Position Embeddings, Position IDs, and Attention Masks maintained after removing the KV Cache entries?

  4. Comparison with other methods: How does the proposed approach compare with existing visual token pruning or KV cache compression methods in terms of the Efficiency–Accuracy Trade-off?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.