- Rating
- 4(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper focuses on reducing the inference-time visual computational overhead in MLLMs. The paper conducts probing experiments and empirically demonstrates that most of the visual information can be simply retained via MLPs, rather than visual queries, visual attention outputs, or visual feed-forward updates. Motivated by this, the paper proposes -Vision, which trains MLP adapters at each layer to retain visual information without knocking out any visual tokens, and maintains visual memory for text interactions. The empirical results suggest that the proposed methods can reduce FLOPS while surpassing other pruning-based methods on various benchmarks.
Critical strengths
- The paper provides an interesting insight into visual tokens in MLLMs: the high-cost attention operations on visual tokens are not necessary in some cases and can be replaced by more efficient operations like MLP projections.
- The proposed method is simple and empirically effective. Replacing the visual computational overhead with simple MLPs is easy to understand, according to recent insights in [1].
- The paper conducts experiments in various scenarios to demonstrate the effectiveness of the proposed method and conducts many ablation studies to inspect the mechanism of the method.
- The paper is well-written, making it easy to follow.
[1] ShortV: Efficient Multimodal Large Language Models by Freezing Visual Tokens in Ineffective Layers. (ICCV 2025)
Critical weaknesses
- As the paper argues that we should reduce visual computation rather than reducing tokens themself, it lacks comparison with the same kind of methods. Only the pruning-based methods are compared in the main experiments. To my knowledge, ShortV [1] also works on reducing visual computation by freezing computation at some layers. Comparison with the same kind of methods will further distinguish the contribution of this paper.
- Motivating the method by analysing the low-rank nature of visual hidden states doesn't seem to be very convincing to me. The low-rank itself doesn't necessarily guarantee that the residuals of visual hidden states can be predicted by MLPs. The experiments in Figure 1(b) show that the cosine similarity can decline to about 0.7, which is pretty low, I think, given the high dimensionality of the hidden states. Therefore, I don't think MLPs can maintain the visual information well in this case.
- Turning off the DeepStack of Qwen acts as a risk, making the evaluation differ from what the model is originally trained for. The authors can provide results with the DeepStack on to fully show the effects of the method.
Questions for authors
- What's the effect of -Vision against other similar methods such as ShortV? Comparison within the same method family can further distinguish the method.
- Can the method be motivated by a more straightforward empirical analysis? The low-rank analysis is currently not strongly related to the method.
- What will the results change if the DeepStack is left on? I know it may cost a lot if the main results are rerun with it. The authors can instead provide more fine-grained analysis on it.
Dear Reviewers,
Please carefully review the authors’ rebuttal and engage in further discussion with the authors to clarify any remaining concerns and questions. Your active participation during the discussion period is greatly appreciated.
Best, AC