acceptodds
Withdrawn from ICLR 2027

Same Scene, Different Decisions: Material Sensitivity in Embodied Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Embodied models must connect visual appearance to task-relevant decisions. We study their material sensitivity: how those decisions change when scene geometry is retained but materials are converted to a different representation. In 99 paired navigation episodes, success falls from 52.5% to 48.5%, yet 30 episodes change success status: gains and losses largely cancel in the average. We investigate this hidden turnover through paired visual grounding and navigation experiments. A readout study across six vision-language model configurations separates localization from response-format and coordinate-scale effects. Controlled two-view tests then require models to follow a queried object across positions, and target-window crossovers reveal that changes to the target and its surroundings can shift grounding in opposite directions. In navigation, 17 previously successful episodes fail while 13 previously failed episodes succeed under converted materials. These results put the identity of successful decisions alongside mean accuracy as a central object of evaluation. Material representation is therefore more than an asset-format choice: it is an input variable through which we can study the visual dependence of embodied models.

Primary area:
applications to robotics, autonomy, planning
Keywords:
vision-language models, visual grounding, embodied navigation, material sensitivity, material conversion, controlled visual interventions

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_51hMOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The authors tackle an interesting problem: How is visual perception reliant on the material properties of the object / scene of interest? Through paired interventions, where an original scene is altered by modifying only the materials and not the geometry, the authors show that the decisions made by VLMs for grounding (i.e., localizing objects) and navigation (reaching a specified image target) vary between the original and modified scenes. The tested models include Gemma4-E4B, Qwen2.5-VL-7B, Qwen3-VL-2B, Qwen3-VL-8B, InternVL3-8B and Molmo-7B-D. The results demonstrate modest variation in grounding performance and larger variation in navigation performance based on material interventions. The authors highlight the importance of paired interventions with decision-flip analysis, and the need to focus on material properties, not just geometry for embodied tasks.

Critical strengths
  • The paper focuses on an interesting problem: How does material property affect VLM visual perception, especially in the context of navigation tasks?
  • The intervention of varying material properties while keeping the rest of the scene fixed (including the object colors) is scientifically sound and useful.
Critical weaknesses

I have three key weaknesses for the paper.

1. Novelty

There is a large body of prior work that has studied the importance of appearance and material properties for visual perception in neural networks [1,2,3] and VLMs [4,5,6,7,8]. These have not been cited nor compared with. The motivation for this particular study and why the results are surprising / unexpected is therefore unclear. There's also relevance of adversarial perturbations [9,10] (e.g., are the material variations observed now a form of adversarial perturbation?).

[1] Geirhos, Robert, et al. "ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness." arXiv preprint arXiv:1811.12231 (2018). [2] Hendrycks, Dan, and Thomas Dietterich. "Benchmarking neural network robustness to common corruptions and perturbations." arXiv preprint arXiv:1903.12261 (2019). [3] Zhang, Chenshuang, et al. "Imagenet-d: Benchmarking neural network robustness on diffusion synthetic object." 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2024. [4] Gavrikov, Paul, et al. "Can we talk models into seeing the world differently?." International Conference on Learning Representations. Vol. 2025. 2025. [5] Eppel, Sagi, Mor Bismut, and Alona Strugatski. "Shape and texture recognition in large vision-language models." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2026. [6] Wang, Jiabao, et al. "The Constant Eye: Benchmarking and Bridging Appearance Robustness in Autonomous Driving." arXiv preprint arXiv:2602.12563 (2026). [7] Burgert, Tom, et al. "ImageNet-trained CNNs are not biased towards texture: Revisiting feature reliance through controlled suppression." Advances in Neural Information Processing Systems 38 (2026): 60809-60830. [8] Sar, Ayan, et al. "VLM Reality Check: A Causal Counterfactual Benchmark for Diagnosing Cognitive Biases in Vision-Language Models." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2026. [9] Chaubey, Ashutosh, et al. "Universal adversarial perturbations: A survey." arXiv preprint arXiv:2005.08087 (2020). [10] Tong, Shengbang, et al. "Eyes wide shut? exploring the visual shortcomings of multimodal llms." 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2024.

2. Poor experimental setup

The experimental setup has several critical flaws.

  • The number of samples is so low (99 for navigation --- L323, 16 queries per intervention for grounding --- Table 1) that the results can be noisy and unreliable.
  • The null-hypothesis of re-evaluating models on the original stimulus and measuring performance / flips is not performed clearly (Tables 1 and 2). This is hinted at in L311 for grounding, where the decoding variability on its own explains ~16.7% of the cases in Qwen3-VL-2B vary without any changes to the stimuli. Is the variability seen just generation / low-data regime noise?
  • Alternate hypotheses for the variability in model predictions are not evaluated. E.g., if the color were changed while holding the materials fixed, or if the background was changed while keeping the objects fixed, how much would the performance vary? This type of variability could be likely due to models being weak, which is likely here.
  • All models evaluated are small / relatively old - Gemma4-E4B, Qwen2.5-VL-7B, Qwen3-VL-2B, Qwen3-VL-8B, InternVL3-8B and Molmo-7B-D. Each of these models have much better alternatives from the recent year, and also have higher capacity variants. None of the closed-source SoTA models like Claude Opus, Gemini, and GPT 5-* are evaluated. Given the rapid progression in model capacities, the findings could just be outdated.

Overall, the results are quite inconclusive and I don't have any good takeaways at the end of the paper due to the above concerns.

3. Paper writing clarity is poor.

  • The abstract is poorly written, with no indication of the significance of the results (e.g., navigation results can flip even normally across trials without any interventions for the same model and scene).
  • The introduction does not motivate the problem at all (why is it important to solve?). What has been done so far in this problem space and what does this paper contribute that is new?
  • Figure 1 is hard to understand. How are the middle and right panels related to material change? The original vs. swapped / converted do not convey a material change. The scene itself seems to have changed.
  • The experimental findings are not clearly analyzed. Examples:
    • L212 - 237 --- what is the takeaway? 0 - 3 flips out of 16 across conditions. Why is this meaningful?
    • L308-312 --- how does decoding variability (and more generally, generation variability) impact the conclusions?
Questions for authors
  1. Why is this work novel?
  2. Are models particularly sensitive to material properties, or are they just sensitive to any visual changes? Is the null-hypothesis rejected?
  3. How do newer and larger models compare? Are these limitations still existing in models? Are the findings outdated?
  4. The paper writing in general is poor, but addressing the key clarity concerns would be helpful.
Reviewer_GRGEOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper appears to describe some focused tests of the ability of computer vision/AI systems to use verbal instructions (e.g., sentences) to make decisions about and navigate 3D rendered visual scenes across modest changes in material properties of the scenes. One prominent argument the paper makes is that overall decision performance may not be an ideal outcome measure, as a variety of specific stimuli result in different decision outcomes (e.g., some go from correct to incorrect with a change in material, while others change in the opposite direction). The paper seems to use relatively few pairs of rendered scenes for analyses and the justification for this choice is not made particularly clear. Details of the navigation experiments are poorly presented, so I cannot comment on this aspect of the work.

Critical strengths
  1. The paper appears to dedicate substantial effort to optimizing the pairs of scenes, though this results in a small number of tested scene pairs
  2. Careful consideration of when model decisions change from correct/incorrect rather than overall changes in accuracy is an important focus that may enable new insights into the specific material property changes that are most impactful, which could guide design of computer vision systems supporting embodied agents
Critical weaknesses
  1. The paper is very challenging to read and understand in its present form. Many of the acronyms are not defined, the rationale for the experiments is only briefly argued, and there is minimal discussion of how the static scene and navigation experiments are related
  2. Navigation experiments in particular are not clearly described, and the data presented are not interpreted in a way that is useful for the broad audience expected for submissions to this venue
  3. It appears that only a very, very narrow set of stimulus pairs is used throughout. Given that these pairs appear to be systematically and automatically generated, it’s unclear why only a small number are chosen. One would expect millions of pairwise comparisons rather than dozens. Though I highlight that characterizing changes to decisions (rather than changes to accuracy overall) is a strength, that strength is largely offset by the extremely narrow set of stimuli used without clear justification
  4. There is no attempt to relate AI system performance in these tasks to human performance, and performance variability is only minimally addressed. Any real-world application of computer vision systems that need to be robust across small input changes would also need to be robust to noise, which is not examined or addressed here
  5. The paper mentions that changes in material properties could result in changes in color, contrast, texture, etc – this seems likely to change the ‘salience’ of different locations in the image, which could itself impact performance of computer vision systems
Questions for authors
  1. What is the potential role of differences in object salience on the results? Can this be experimentally determined (or ruled out)?
  2. Why are only a small set of image pairs chosen?

My assessment could only be materially improved through substantial edits to the paper to improve its clarity and flow, clearer explanations of the experiments and results, and inclusion of reasonable justifications of design/experiment choices addressed in Weaknesses above.

Reviewer_zAz6OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies if changing the material representation of a 3d scene affects the decision of an embodied model while keeping the scene geometry fixed.

The authors investigate this through visual grounding experiments with several VLMs and closed-loop navigation experiments.

The main finding is that the overall aggregate performance can hide substantial changes in individual decisions. The grounding experiment further examine how output formats, object position, and changes to target affect the presentation.

Critical strengths
  1. this is an interesting probelm with strong prctical releveance. Different environments may have the same geometric structure but different materials. The consistency of embodied models across these enviorment is essential to the success the of the task,

  2. some good eveluation provide usefull information beyond average accuracy, the controlled grounding experiments also attempt to separate different sources of variation.

Critical weaknesses
  1. The biggest concern for this paper is that it primarily just observes changes in model decisions without proposing any solution to address the material sensitivity for embodied model or providing a deeper explanation of what causes these sensitivities. The main finding that individual predictions can change despite similar aggregate performance is somewhat expected. While this observation is interesting, there are limited methodological or theoretical insights. In my opinion, the novelty and significance of the contribution fall well below the bar for an ICLR-level conference.

  2. The paper is also very difficult to follow, with several unconnected experiments and insufficient explanation of how the contributions relate to the central RQ of this paper. for example, sections 4 and 5 switch between material sensitivity, coordinate readout, and output format without clearly explaining how they relate. The connection between the grounding experiments and the downstream navigation results is also not well established.

  3. Grounding experiments use only four source pairs, and navigation experiments lack repeated runs under identical conditions; there is no strong stats evidence to distinguish material sensitivity from runtime variability.

Questions for authors
  1. The grounding experiments only use four pairs. Would the findings still hold with more objects and scenes? What caused such a small scale of experiments?

  2. How much of the reported outcome change could simply come from randomness in the navigation process?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.