- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper explores if there is more information than just object categories, such as instance-level spatial relationship information in VLMs. The authors use three model families (Qwen3VL, LLaVA-OneVision, LLaMa-3.2-vision) to train low-rank linear mapping that maps the representations from the vision encoder and LLM backbones. They evaluate the performance on several benchmarks to show the instance spatial level information exists. They also do analysis by using representation similarity, semantic mismatching, concept swapping to show the mapping recovers cross-modal structure, which shows the language representation can retrieve matching visual patches.
Critical strengths
1.The question is clearly defined. The paper use 4 levels to explain the results to distinguish “decodable information” from “information actually used by the model”. 2. The are extensive controllable experiments to rule out the trivial explanations, including geometry analysis, semantic mismatching, concept swapping and so on. 3. Mapping-guided reweighting improve the several downstream tasks performance, including v*, ADE20K Count, POPE random.
Critical weaknesses
- The authors argues about the language states “remember a specific instance”. However, the localization jointly use the language representations and image patches instead of using the using language representation itself. It’s more like language representation include the instance-level conditioned visual query information.
- The baseline comparison is not clear. The authors don’t provide the experimental setup for the baselines like CLIP, groundingDINO. Since the mapping in paper is supervised trained, it will be unfair to compare the CLIP directly.
- There is no explanation for the high pointing accuracy but low IoU scores.
- The pipeline is not clear enough. If the image and text pair is the input, then the text token will also have the information based on the self-attention from the image tokens. It will be better to compare language representation itself only (no image input) as the query.
- It will be better to have an ablation study for the shuffled image patches with text representations.
- Missing related work.
- Li, Jiaang, et al. "Seeing or Knowing? Visual Context Sensitivity in Multimodal Large Language Models."
- Huh, Minyoung, et al. "The platonic representation hypothesis."
- Li, Jiaang, et al. "Do vision and language models share concepts? a vector space alignment study."
- Esmaeilkhani, Parsa, and Longin Jan Latecki. "Preserving Localized Patch Semantics in VLMs."
Questions for authors
See weaknesses.