- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates visual-token usage across generation time and transformer depth on multimodal reasoning tasks. Experiments on four Qwen3-VL model sizes show that visual attention concentrates early in generation and in intermediate layers. The authors then remove visual KV-cache entries at selected decoding steps or upper layers, finding that some direct visual access can be removed with limited accuracy degradation. The appendices provide analytical estimates of computation and cache savings.
Critical strengths
- Jointly examining generation time and transformer depth provides a useful perspective on visual access during reasoning.
- The paper reports task-dependent accuracy-reduction trade-offs and provides an analysis of attention computation and KV-cache savings.
Critical weaknesses
- The incremental contribution over prior visual-attention studies is limited. Prior work already investigates layer-dependent visual-token redundancy and motivates reducing visual processing in later layers. The paper acknowledges this literature, but its main distinction is the focus on long-form multimodal reasoning and complete visual-token removal. This is a useful extension, yet the additional insight needs clearer substantiation.
- Practical runtime and inference-engine compatibility are not demonstrated. Appendix B provides analytical FLOP and KV-cache estimates, but no measured end-to-end latency, throughput, or peak GPU memory. Logical cache reduction does not automatically translate into equivalent practical savings. The authors should clarify compatibility with optimized engines such as vLLM, particularly for layer-specific cache lengths and mid-generation eviction, and measure speed and memory against the unmodified engine.
- The efficiency benefit at near-baseline accuracy appears modest and task-dependent.
- Generalization across model architectures remains unverified. All experiments use Qwen3-VL. Scaling within one family does not establish that the temporal integration window or depth-wise pattern generalizes across architectures. An additional family, such as Gemma 4 12B, would strengthen the findings.
- Multi-turn dialogue may invalidate the assumption behind permanent visual-cache removal. The experiments examine continued generation for a fixed question. In multi-turn dialogue, a later question may concern a different object or previously unused detail, which representations accumulated for the first response may not preserve. A controlled second-turn evaluation with substantially different visual questions would clarify this limitation.
Questions for authors
How much post-removal robustness comes from image-conditioned prompt-token states versus previously generated reasoning-token states? Can these sources be separated to support the proposed explanation?