- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The authors tackle an interesting problem: How is visual perception reliant on the material properties of the object / scene of interest? Through paired interventions, where an original scene is altered by modifying only the materials and not the geometry, the authors show that the decisions made by VLMs for grounding (i.e., localizing objects) and navigation (reaching a specified image target) vary between the original and modified scenes. The tested models include Gemma4-E4B, Qwen2.5-VL-7B, Qwen3-VL-2B, Qwen3-VL-8B, InternVL3-8B and Molmo-7B-D. The results demonstrate modest variation in grounding performance and larger variation in navigation performance based on material interventions. The authors highlight the importance of paired interventions with decision-flip analysis, and the need to focus on material properties, not just geometry for embodied tasks.
Critical strengths
- The paper focuses on an interesting problem: How does material property affect VLM visual perception, especially in the context of navigation tasks?
- The intervention of varying material properties while keeping the rest of the scene fixed (including the object colors) is scientifically sound and useful.
Critical weaknesses
I have three key weaknesses for the paper.
1. Novelty
There is a large body of prior work that has studied the importance of appearance and material properties for visual perception in neural networks [1,2,3] and VLMs [4,5,6,7,8]. These have not been cited nor compared with. The motivation for this particular study and why the results are surprising / unexpected is therefore unclear. There's also relevance of adversarial perturbations [9,10] (e.g., are the material variations observed now a form of adversarial perturbation?).
[1] Geirhos, Robert, et al. "ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness." arXiv preprint arXiv:1811.12231 (2018). [2] Hendrycks, Dan, and Thomas Dietterich. "Benchmarking neural network robustness to common corruptions and perturbations." arXiv preprint arXiv:1903.12261 (2019). [3] Zhang, Chenshuang, et al. "Imagenet-d: Benchmarking neural network robustness on diffusion synthetic object." 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2024. [4] Gavrikov, Paul, et al. "Can we talk models into seeing the world differently?." International Conference on Learning Representations. Vol. 2025. 2025. [5] Eppel, Sagi, Mor Bismut, and Alona Strugatski. "Shape and texture recognition in large vision-language models." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2026. [6] Wang, Jiabao, et al. "The Constant Eye: Benchmarking and Bridging Appearance Robustness in Autonomous Driving." arXiv preprint arXiv:2602.12563 (2026). [7] Burgert, Tom, et al. "ImageNet-trained CNNs are not biased towards texture: Revisiting feature reliance through controlled suppression." Advances in Neural Information Processing Systems 38 (2026): 60809-60830. [8] Sar, Ayan, et al. "VLM Reality Check: A Causal Counterfactual Benchmark for Diagnosing Cognitive Biases in Vision-Language Models." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2026. [9] Chaubey, Ashutosh, et al. "Universal adversarial perturbations: A survey." arXiv preprint arXiv:2005.08087 (2020). [10] Tong, Shengbang, et al. "Eyes wide shut? exploring the visual shortcomings of multimodal llms." 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2024.
2. Poor experimental setup
The experimental setup has several critical flaws.
- The number of samples is so low (99 for navigation --- L323, 16 queries per intervention for grounding --- Table 1) that the results can be noisy and unreliable.
- The null-hypothesis of re-evaluating models on the original stimulus and measuring performance / flips is not performed clearly (Tables 1 and 2). This is hinted at in L311 for grounding, where the decoding variability on its own explains ~16.7% of the cases in Qwen3-VL-2B vary without any changes to the stimuli. Is the variability seen just generation / low-data regime noise?
- Alternate hypotheses for the variability in model predictions are not evaluated. E.g., if the color were changed while holding the materials fixed, or if the background was changed while keeping the objects fixed, how much would the performance vary? This type of variability could be likely due to models being weak, which is likely here.
- All models evaluated are small / relatively old - Gemma4-E4B, Qwen2.5-VL-7B, Qwen3-VL-2B, Qwen3-VL-8B, InternVL3-8B and Molmo-7B-D. Each of these models have much better alternatives from the recent year, and also have higher capacity variants. None of the closed-source SoTA models like Claude Opus, Gemini, and GPT 5-* are evaluated. Given the rapid progression in model capacities, the findings could just be outdated.
Overall, the results are quite inconclusive and I don't have any good takeaways at the end of the paper due to the above concerns.
3. Paper writing clarity is poor.
- The abstract is poorly written, with no indication of the significance of the results (e.g., navigation results can flip even normally across trials without any interventions for the same model and scene).
- The introduction does not motivate the problem at all (why is it important to solve?). What has been done so far in this problem space and what does this paper contribute that is new?
- Figure 1 is hard to understand. How are the middle and right panels related to material change? The original vs. swapped / converted do not convey a material change. The scene itself seems to have changed.
- The experimental findings are not clearly analyzed. Examples:
- L212 - 237 --- what is the takeaway? 0 - 3 flips out of 16 across conditions. Why is this meaningful?
- L308-312 --- how does decoding variability (and more generally, generation variability) impact the conclusions?
Questions for authors
- Why is this work novel?
- Are models particularly sensitive to material properties, or are they just sensitive to any visual changes? Is the null-hypothesis rejected?
- How do newer and larger models compare? Are these limitations still existing in models? Are the findings outdated?
- The paper writing in general is poor, but addressing the key clarity concerns would be helpful.