acceptodds
Withdrawn from ICLR 2027

ReMo: Rethinking Modality Roles in Vision-Language Models for General Visual Perception and Reasoning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Despite remarkable strides in multimodal understanding, the general visual perception of Vision-Language Models (VLMs) remains a critical weakness. Existing solutions typically rely on resource-heavy knowledge distillation or latency-intensive external visual tools, neither of which fundamentally improves the VLM's intrinsic perceptual mechanisms. In this paper, we present , a novel framework that thinks the complementary dality roles of language and vision to achieve intrinsic visual enhancement in VLMs. Our core insight is that language should actively orchestrate visual perception rather than merely respond to it. ReMo first leverages the expansive reasoning capabilities of Large Language Models (LLMs) to generate linguistic priors, dynamically extracting reasoning-aware guidance from cross-modal attention to localize task-relevant visual evidence. However, since cross-modal alignment often discards fine-grained visual details, we introduce a structural complementary mechanism. By exploiting the innate relationship modeling of Vision Transformers (ViTs), ReMo actively recovers vital visual contexts that are typically overlooked by the LLM. By jointly harmonizing language-guided visual search and ViT-driven visual recovery alongside visual evidence reinforcement, our framework significantly elevates the general visual perception and reasoning capacities of VLMs.

Primary area:
foundation or frontier models, including LLMs
Keywords:
VLM, Understanding‑Enhancement

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_PxnROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a training-free framework, named ReMo, to improve visual perception and reasoning in vision-language models by combining language-guided localization, ViT-guided context completion, and attention reweighting. The method is evaluated on Qwen3-VL-4B/8B and Kimi-VL-16B-A3B across high-resolution perception, general visual understanding, and hallucination benchmarks. The reported results show improvements over the corresponding base models. Progressive ablations and qualitative visualizations provide evidence for the contributions of the proposed components.

Critical strengths
  1. The paper reports consistent improvements across multiple VLM backbones and diverse evaluation settings. Experiments cover both dense and MoE models, with gains on high-resolution perception, general visual understanding, and hallucination benchmarks.
  2. The language-guided localization and ViT-guided context completion is well motivated. Task-relevant language cues identify candidate evidence, while visual affinities help expand coverage to surrounding context. The framework implements this interaction without updating model parameters and preserves the full visual token sequence during attention modulation. This provides a useful approach to exploiting complementary information already available within pretrained VLMs.
Critical weaknesses

1. The novelty and advantages over closely related methods are insufficiently demonstrated.

For language-guided localization, internal attention-based localization and semantic guidance already appear in MLLMs Know Where to Look [1] and VGA [2], while VAALE [3] combines generated textual context with attention refocusing. The additional contribution of ReMo’s knowledge priors and semantic-anchor denoising requires clearer justification. For ViT-guided completion, VEGAS [4] already uses vision-encoder attention to guide the language model. AnchorPrune [5] supplements query-relevant tokens with complementary context, and HiPrune [6] preserves neighboring tokens around important regions. Although the latter two target pruning, their context-selection strategies remain relevant to ReMo’s claimed novelty. For evidence reweighting, training-free visual attention enhancement is established by PAI [7], VAR [8], and VGA [2]. The paper should clarify what its specific combination contributes beyond these precedents.

The experiments also omit these methods and primarily compare against base VLMs. Representative attention-enhancement baselines should be evaluated using the same backbone and protocol, while alternative context-selection strategies should be compared under controlled token budgets. Without such discussion and comparisons, the results demonstrate improvements over base models but do not varify the novelty or advantages of its proposed combination.

2. The experiments do not isolate the value of ViT-guided completion or support the claimed efficiency advantages.

The progressive ablation in Fig. 4 is useful, but adding ViT-guided completion also increases the number of visual tokens receiving an attention boost. Without comparisons that enhance the same number of tokens, such as spatial expansion or selection by the original relevance scores, it remains unclear whether the gains come from ViT affinities or simply from covering more visual content. Furthermore, the efficiency argument is not supported by end-to-end latency or memory measurements, though the additional 4B VLM is only used for prior generation.

Reference

[1] Khayatkhoei, Mahyar, Prateek Chhikara, and Filip Ilievski. "Mllms know where to look: Training-free perception of small visual details with multimodal llms." International Conference on Learning Representations. Vol. 2025. 2025.

[2] Zhao, Jianfei, et al. "Tell model where to look: Mitigating hallucinations in mllms by vision-guided attention." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2026.

[3] Wang, Jingyi, Fei Li, and Rujie Liu. "Attention to details, logits to truth: visual-aware attention and logits enhancement to mitigate hallucinations in LVLMs." ICASSP 2026-2026 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2026.

[4] Wang, Zihu, et al. "VEGAS: Mitigating Hallucinations in Large Vision-Language Models via Vision-Encoder Attention Guided Adaptive Steering." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2026.

[5] Oh, Kyuan, and Bumsoo Kim. "AnchorPrune: Relevance-Anchored Contextual Expansion for Visual Token Pruning." European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2026.

[6] Liu, Jizhihui, et al. "HiPrune: Hierarchical Attention for Efficient Token Pruning in Vision-Language Models." Findings of the Association for Computational Linguistics: ACL 2026. 2026.

[7] Liu, Shi, Kecheng Zheng, and Wei Chen. "Paying more attention to image: A training-free method for alleviating hallucination in lvlms." European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2024.

[8] Kang, Seil, et al. "See what you are told: Visual attention sink in large multimodal models." International Conference on Learning Representations. Vol. 2025. 2025.

Questions for authors
  1. Novelty and comparative advantage. What specific technical advances distinguish ReMo’s semantic-anchor refinement, ViT-guided completion, and attention biasing from the related methods discussed above? Can the authors provide a comparison with these representative attention-enhancement methods using the same backbone and evaluation protocol to substantiate ReMo’s advantages?

  2. Contribution of ViT-guided completion. Does the improvement in Fig. 4 persist when the seed set, number of reinforced tokens, and attention-bias magnitude are held fixed across completion strategies?

  3. End-to-end efficiency. What are ReMo’s end-to-end latency and peak memory usage relative to the base VLM, including prior generation by the auxiliary 4B VLM? Can the authors separate the costs of prior generation and subsequent inference, and clarify which measurements support the claimed efficiency advantages?

Reviewer_QEqCOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes ReMo, a framework to improve visual perception and reasoning on LVLMs. It utilises smaller scale VLMs to identify language prior and task relevant scope, and ViT to recover and emphasize task-relevant visual information. The proposed method shows promising results on benchmarks such as MME, POPE, and some high resolution images benchmark datasets.

Critical strengths
  1. The method is well-motivated. The idea of adding language prior and emphasizing task relevant visual information aligns with how a human would answer a question. This is an interesting direction for VLM visual reasoning enhancement.
  2. Consistent improvement across architectures, tasks and in ablation studies. The authors conducted experiments on Qwen and Kimi series models, across multiple vision-dependent benchmark datasets. Results showed significant improvements, promising evidence on the generalizability of the proposed method.
  3. Useful qualitative analysis and ablation. Some qualitative examples are shown in Appendix, showing the effectiveness of localizing task-relevant region step by step.
Critical weaknesses
  1. Primary concern centers around computational efficiency. It is mentioned that efficiency is a gap from existing training-based methods, but this paper did not compare its performance with other training-free methods. The author should report runtime of the proposed method and compare it against other training-free methods. The runtime should include the use of 4B VLMs in stage 1 which I think is computationally heavy for a training-free method.
  2. Following (1), the novelty of this work needs strong justification, given that there are many existing training-free works focusing on steering-based and contrastive-decoding-based methods, see [1-3].
  3. The ablation study does not sufficiently isolate the contribution of different components. For instance, localization and cosine denoising are introduced together, and attention reweighting are not evaluated independently. These experiments are required to show the necessity of extra steps, especially when this is a training-free framework, where runtime is crucial.

Refs: [1] Mitigating Hallucinations in Vision-Language Models through Image-Guided Head Suppression, EMNLP 2025 [2] The Hidden Life of Tokens: Reducing Hallucination of Large Vision-Language Models Via Visual Information Steering, ICML 2025 [3] Same Attention, Different Truths: Put Logit-Lens over Visual Attention to Detect and Mitigate LVLM Object Hallucination, CVPR 2026

Questions for authors
  1. What is the rationale of generating T_prior and T_anchor using full inputs (i.e., image and text)? Language prior usually means something that can be inferred from text alone, and semantic anchor defined in this paper seems to be an information that can be extracted from the text input alone too.
  2. Can the authors provide details about the runtime? For example, GPU memory peak (per stage) and total runtime compared to vanilla methods and the abovementioned methods/line of work?
Reviewer_V3yoOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ReMo, a training-free inference-time framework that improves visual perception in VLMs by combining language-guided localization with ViT-based spatial completion. ReMo uses task-specific linguistic priors and intermediate-layer cross-modal attention to locate potentially relevant image regions. It then refines the localization scores using the similarity between visual tokens and semantic anchors, and uses the VLM’s vision encoder to recover nearby visual evidence that the initial attention map may have missed. During generation, ReMo adds a positive bias to the attention logits of the selected evidence tokens, encouraging the model to attend to them more strongly. Experiments across multiple VLM backbones show performance gains without updating model parameters.

Critical strengths
  • ReMo improves VLM performance at inference time without updating model parameters and is evaluated across multiple model backbones.

  • It reports consistent gains on high-resolution benchmarks, including 4.210.5 percentage points on V*Bench and 5.98.7 percentage points on HR-Bench-8K.

  • The method has a clear progression: language-guided localization identifies candidate regions, ViT attention expands the selected tokens, and attention bias emphasizes them during generation.

Critical weaknesses
  1. On evaluation and effectiveness of components
  • The comparison with a Thinking variant is limited. The paper does not compare ReMo with CoT or crop- and zoom-based visual reasoning methods under matched inference budgets.
  • The paper does not report end-to-end latency, throughput, or memory use, despite requiring an additional 4B VLM call and attention analysis. Its efficiency claims are therefore difficult to assess.
  • Answer accuracy does not establish whether the selected evidence tokens correspond to the regions needed to answer the question. The bounding-box annotations in V*Bench could support a direct localization evaluation.
  • If the initial attention map selects the wrong region, ViT-based expansion may propagate that error and the attention bias may further emphasize irrelevant tokens. The paper does not report how often this occurs or how it affects answers.
  • The ablation does not fully isolate the effect of ViT-based expansion. The localization and completion variants both use attention reweighting, and there is no control that selects the same number of tokens through random or simple local expansion.
  1. On methodology
  • ReMo does not increase image resolution or create new visual features. It cannot recover a tiny object whose information was already lost by the vision encoder. Without results broken down by object size, the contributions of semantic priors, token expansion, and attention reweighting on small-object cases remain unclear.

  • What is the intuition behind aggregating across the full text rather than emphasize relevant words or the extracted semantic anchors? Also, is there any reliability issue on VLM's cross-modal attention map itself? The attention map visualization on appendix doesn't seem to show spatially well-aligned attention activation even in the middle layers.

Questions for authors

None

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.