acceptodds
Under review as a conference paper at ICLR 2027

Just MLPs: Efficient Visual State Reconstruction for Multimodal Language Models

12 views

Abstract

Long visual token sequences often account for a substantial fraction of the computational overhead in multimodal large language models (MLLMs). Existing approaches reduce this cost by pruning redundant visual tokens, but permanently discard visual evidence that may become useful in subsequent layers. We instead ask whether all visual tokens can be preserved while reducing the cost of repeatedly evolving the representations through the Transformer. To answer this question, we perform low-rank interventions on visual-to-text information flow. We find that, after visual-to-text attention is blocked, restoring only a few directions recovers most of the lost accuracy, suggesting the relevant visual influence is concentrated in a low-dimensional subspace. We further observe strong predictability in layer-specific visual states: lightweight MLPs approximate them with high cosine similarity and low reconstruction error. Motivated by these findings, we propose -Vision, which replaces repeated Transformer evolution of visual tokens with lightweight low-rank adapters that construct layer-wise visual memories while preserving all visual tokens for text retrieval. Across image and video benchmarks, -Vision achieves higher accuracy than visual token pruning baselines at comparable or lower computation, while delivering competitive inference efficiency without discarding visual tokens.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multimodal Large Language Models

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_HKCPOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper focuses on reducing the inference-time visual computational overhead in MLLMs. The paper conducts probing experiments and empirically demonstrates that most of the visual information can be simply retained via MLPs, rather than visual queries, visual attention outputs, or visual feed-forward updates. Motivated by this, the paper proposes δ\delta-Vision, which trains MLP adapters at each layer to retain visual information without knocking out any visual tokens, and maintains visual memory for text interactions. The empirical results suggest that the proposed methods can reduce FLOPS while surpassing other pruning-based methods on various benchmarks.

Critical strengths
  1. The paper provides an interesting insight into visual tokens in MLLMs: the high-cost attention operations on visual tokens are not necessary in some cases and can be replaced by more efficient operations like MLP projections.
  2. The proposed method is simple and empirically effective. Replacing the visual computational overhead with simple MLPs is easy to understand, according to recent insights in [1].
  3. The paper conducts experiments in various scenarios to demonstrate the effectiveness of the proposed method and conducts many ablation studies to inspect the mechanism of the method.
  4. The paper is well-written, making it easy to follow.

[1] ShortV: Efficient Multimodal Large Language Models by Freezing Visual Tokens in Ineffective Layers. (ICCV 2025)

Critical weaknesses
  1. As the paper argues that we should reduce visual computation rather than reducing tokens themself, it lacks comparison with the same kind of methods. Only the pruning-based methods are compared in the main experiments. To my knowledge, ShortV [1] also works on reducing visual computation by freezing computation at some layers. Comparison with the same kind of methods will further distinguish the contribution of this paper.
  2. Motivating the method by analysing the low-rank nature of visual hidden states doesn't seem to be very convincing to me. The low-rank itself doesn't necessarily guarantee that the residuals of visual hidden states can be predicted by MLPs. The experiments in Figure 1(b) show that the cosine similarity can decline to about 0.7, which is pretty low, I think, given the high dimensionality of the hidden states. Therefore, I don't think MLPs can maintain the visual information well in this case.
  3. Turning off the DeepStack of Qwen acts as a risk, making the evaluation differ from what the model is originally trained for. The authors can provide results with the DeepStack on to fully show the effects of the method.
Questions for authors
  1. What's the effect of δ\delta-Vision against other similar methods such as ShortV? Comparison within the same method family can further distinguish the method.
  2. Can the method be motivated by a more straightforward empirical analysis? The low-rank analysis is currently not strongly related to the method.
  3. What will the results change if the DeepStack is left on? I know it may cost a lot if the main results are rerun with it. The authors can instead provide more fine-grained analysis on it.
Reviewer_W89WOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes δ-Vision, a method for reducing computational cost while retaining all visual tokens. It uses lightweight low-rank adapters to construct layer-specific visual representations, replacing the standard Transformer updates for visual tokens.

δ-Vision introduces two variants: an embedding adapter that predicts each layer’s visual memory directly from the initial visual embeddings, and a recurrent adapter that updates the memory sequentially across layers.

Experiments across LLaVA and Qwen backbones on single-image, multi-image, and video benchmarks demonstrate the effectiveness of the approach. Compared with the evaluated compression baselines, δ-Vision achieves a favorable trade-off between task performance and computational cost.

Critical strengths
  1. The method is simple and demonstrates useful for replacing expensive visual computation. δ-Vision separates visual-state construction from textual access to visual information. By training lightweight adapters, it removes substantial visual-update computation while retaining access to all visual tokens.
Critical weaknesses
  1. The main accuracy gains and measured efficiency gains are not sufficiently established under the same evaluation conditions. The largest accuracy improvements are demonstrated on single-image benchmarks, whereas the main FLOPs, latency, and memory measurements are reported on Video-MME. Reading Tables 2 and 4 together, the multi-image/video-trained δ-Vision variant achieves 51.0 on Video-MME, compared with 51.3 for DivPrune. Both methods achieve a 1.30× overall inference speedup, while their peak memory usage is 12.97 GB and 10.60 GB, respectively. δ-Vision has a slightly higher prefill speedup, but this setting does not establish a clear overall deployment advantage. Reporting actual latency and memory alongside the single-image accuracy gains would make the claimed accuracy–efficiency trade-off substantially more convincing.

  2. There is an evidential gap between the low-rank analyses and the final visual-memory predictor. The reconstruction experiment in Figure 1(b) uses a full-width MLP, whereas the proposed method uses a low-rank residual adapter. The predictability demonstrated by the former does not directly establish the reconstruction capability of the latter. Moreover, the final adapter is trained using output-distribution KL distillation, and the paper does not directly show how closely its predicted memories match the teacher’s original visual states. The experiments therefore provide stronger support for learning useful substitute visual context than for accurately reconstructing the original visual-state trajectory.

Questions for authors
  1. explain the experiments setting of Table1, especially rank r. Could you report accuracy, absolute latency, and peak memory together for δ-Vision and the strongest pruning baseline under the main single-image setting in Table 1, specifying input and output lengths?
  2. How are the projection bases in Figure 3 and Table 6 constructed? Are they fitted on independent data and fixed during evaluation, or recomputed for each test example? What are the dimensions of the decomposed matrices, which tokens or queries do they contain, and are samples aggregated? These details are necessary to interpret the recovery obtained at rank 128.
  3. How was the default rank of 128 selected? Please state it explicitly in Section 4.1 and the main table captions.
Reviewer_nPd2OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes δ-Vision to reduce the computational cost of processing visual inputs in MLLMs. Visual inputs are typically represented by long token sequences that require substantial computation at every Transformer layer, and this work aims to reduce this cost with δ-Vision while preserving all visual tokens. The authors find empirical evidence that visual tokens do not necessarily require full layer-wise Transformer computation. Based on this, they propose a method that replaces the layer-wise update of visual tokens with low-rank adapters, so that visual tokens are used only in the key and value computation attended by text queries. The method reduces the computational budget, shows inference speedup, and outperforms previous visual token pruning methods at comparable computational cost.

Critical strengths
  • The proposed method could be employed with FlashAttention and other efficient LLM serving libraries such as vllm and SGLang, because the method does not manipulate attention maps or prune visual tokens differently at each layer (so the method preserves the standard parallel computation of MLLMs) and only adds low-rank adapters to reduce the computational cost of visual tokens. I think it would be better to show a latency analysis under such backends, since it is a clear advantage for practical usage in industry.
  • The findings presented in the paper are interesting, and the method built upon these findings looks promising.
  • While previous methods reduce the computational cost by pruning visual tokens, this paper is notable in that it explores a new axis, reducing the computation on visual embeddings instead of removing visual tokens.
Critical weaknesses
  • In the Preliminaries (lines 117–128), the formulation seems to assume that visual tokens always come before text tokens. In multi-turn or interleaved settings such as [image 1, text, image 2], it is unclear how the method can be applied, so additional clarification would be helpful.
  • Some experimental details are missing. For example, for Table 2, the paper does not mention which GPU, precision, attention backend, or LLM serving backend was used, or how the speed was measured.
  • The paper should discuss ViCA [1] and Fan et al. [2]. Neither of them reduces the number of visual tokens!. Instead, both show that visual tokens do not need the full computation when passing through the LLM Transformer layers. In particular, ViCA looks very similar to the proposed approach, since it also bypasses the self-attention and FFN computation for visual tokens and uses them only as keys and values for text tokens. Clarification on the difference from these works is needed.

[ref1]: ViCA: Efficient Multimodal LLMs with Vision-Only Cross-Attention (https://arxiv.org/abs/2602.07574) [ref2]: What Do Visual Tokens Really Encode? Uncovering Sparsity and Redundancy in Multimodal Large Language Models (CVPR2026)

Questions for authors

For Table 2, which GPU type, batch size, framework, and attention implementation did you use when you measured the numbers?

Comments on the paper

2 comments

Dear Reviewers,

Please carefully review the authors’ rebuttal and engage in further discussion with the authors to clarify any remaining concerns and questions. Your active participation during the discussion period is greatly appreciated.

Best, AC

AuthorsOpenReview ↗

Rebuttal Summary

We sincerely thank the Area Chair and all reviewers for their active engagement during the rebuttal period, their careful consideration of our responses, and their constructive feedback. Below, we provide a brief summary of the main developments and outcomes of the rebuttal.

First, we greatly appreciate the reviewers’ recognition of the following strengths of our work:

  • Interesting insights and a new direction for token efficiency (Reviewer nPd2, Reviewer HKCP).
  • Simple and effective method (Reviewer W89W, Reviewer HKCP).
  • Clear presentation and solid experiments with extensive ablations (Reviewer HKCP).
  • Promising approach with potential for practical deployment (Reviewer nPd2).

Responses to the Main Concerns

In response to the weaknesses and questions identified by the reviewers we made the following clarifications and additions:

  1. Experimental and Implementation Details (Reviewer nPd2, Reviewer W89W): We clarified the handling of interleaved image-text inputs, inference configurations, low-rank projection procedures, and the choice of rank 128. We also provided matched single-image evaluations reporting accuracy, FLOPs, latency, and memory usage.
  2. Comparisons with Closely Related Methods (Reviewer nPd2, Reviewer HKCP): We clarified our differences from the related papers reviewer proposed, and added direct comparisons with ViCA and ShortV, demonstrating stronger accuracy–efficiency trade-offs.
  3. Low-Rank Analysis and Visual Memory Reconstruction (Reviewer W89W, Reviewer HKCP): We conducted additional residual low-rank analyses and MSE-versus-KL reconstruction experiments, providing stronger empirical evidence for our design while clarifying the distinction between hidden-state reconstruction and functional visual-memory prediction.
  4. DeepStack Compatibility (Reviewer HKCP): We evaluated δ-Vision with DeepStack enabled and disabled, including adapter retraining, demonstrating its compatibility with DeepStack and further validating its effectiveness.

Outcomes

The reviewers’ assessments also changed positively following the rebuttal:

  • Reviewer nPd2 thinks most concerns have been resolved and will keep the positive score of 3.
  • Reviewer W89W was satisfied with our response and increased the score from 2 to3.
  • Reviewer HKCP thinks all original concerns are addressed and raise score from 3 to 4.

Overall, we believe that our rebuttal has thoroughly addressed the reviewers' major concerns through additional experiments and detailed clarifications. The new comparisons with related methods, matched accuracy–efficiency evaluations, low-rank and visual-memory analyses, and DeepStack experiments have further strengthened the empirical evidence and technical clarity of our work. We have added these experimental results, methodological clarifications into the latest manuscript as well.

We once again sincerely thank the Area Chair and all reviewers for the time, effort, and thoughtful feedback they devoted to the review and discussion process.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.