acceptodds
Under review as a conference paper at ICLR 2027

How Vision-Language Models Form Ambiguous Depth Interpretations: A Sparse Feature Analysis of Visual Evidence Transformation

Abstract

Vision-language models (VLMs) can produce different depth interpretations of ambiguous images, yet how conflicting visual cues are progressively organized into interpretation-related representations within visual encoders remains poorly understood. Here, we investigate this question using Mach card stimuli containing convex-supporting texture cues and concave-supporting contour cues. We construct controlled stimuli with isolated cues and consistent depth conditions, and analyze sparse visual features across CLIP layers of the LLaVA visual encoder using sparse autoencoders (SAEs). By combining cross-condition feature selection, layer-wise tracking, spatial localization, and causal intervention, we characterize the relationship between visual cue encoding, depth interpretation representations, and model outputs. We find that features associated with specific visual cues emerge in early visual layers, whereas features representing opposing depth interpretations mainly arise in intermediate layers and can be jointly activated under ambiguous conditions. Feature steering and ablation further demonstrate that manipulating these candidate features systematically alters the model’s convex–concave logit margin, with selected contour related interventions shifting the model’s preferred depth interpretation. These findings suggest that ambiguous interpretation in VLMs does not arise from a direct mapping between visual input and final output, but through a hierarchical transformation from visual cue encoding to depth-related interpretation representations. Our study reveals the internal organization of ambiguous visual interpretation in vision-language models and provides a mechanistic framework for understanding how multimodal systems transform competing visual evidence into language-level decisions.

Primary area:
interpretability and explainable AI
Keywords:
Vision-language models, Mechanistic interpretability, Visual ambiguity

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_YAtMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how conflicting texture and contour cues influence depth judgments in a vision–language model. Steering, ablation, and cross-layer restoration and transfer experiments suggest that selected features affect the output margin and that later features carry part of an earlier intervention’s effect.

Critical strengths

The paper addresses a worthwhile question: whether visual information identified in a representation actually contributes to the model’s judgment. Controlled cue conflicts provide a useful setting for investigating this distinction.

The restoration and transfer experiments are a thoughtful addition to feature visualization and ablation. Together, they offer some evidence for local dependencies between selected internal contributions.

Critical weaknesses
  1. The restoration and transfer results provide evidence that selected later contributions can carry part of an earlier perturbation’s effect. What remains unclear is how specific this behavior is to the proposed depth features. The paper does not provide matched random-feature or unrelated-feature interventions with comparable activation levels, perturbation norms, and spatial coverage. A zero-dose control verifies that an unchanged activation yields an unchanged result. It does not establish that the selected direction has a distinctive functional role.

  2. There is also a missing connection between manipulating an image cue and steering an SAE feature. The upstream perturbation is introduced directly into the hidden state, and the paper does not show that it reproduces the internal change caused by modifying the actual texture or contour. Paired image interventions would help establish whether the identified paths participate in processing naturally occurring cue changes.

  3. Averaging semantically reoriented margins produces a numerical score, but the paper has not established that changes in this score reliably reflect changes in concave or convex interpretation. The experiments demonstrate that selected visual-encoder interventions affect the output. Assigning those effects a stable depth meaning requires further validation. I would like to see absolute logits before and after intervention for each mapping, together with alternative prompts or direct semantic answer scoring. An intervention need not reverse the final answer to be informative, but its semantic direction should survive a change in option order.

Questions for authors

See Weaknesses

Reviewer_shpFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates how VLMs interpret visual depth, particularly when conflicting cues suggest concave and convex interpretations of the same image. Using SAEs, the authors analyze the visual representations of LLaVA-1.6-Vicuna-7B to identify depth-cue-related features and examine their causal influence on model outputs. The results show that selected SAE features associated with opposing depth cues can coexist and that intervening on these features can alter output logit margins. However, strong answer-position bias and inconsistent intervention effects across answer mappings raise concerns about whether these findings reflect reliable semantic depth judgments rather than changes in position-sensitive output scores.

Critical strengths
  • The idea of using texture-based convexity and contour-based concavity to investigate conflicting visual cues at the representation level in VLMs is interesting. The use of SAE provides a systematic approach to identifying and analyzing cue-related features within the visual encoder.

  • The paper investigates whether features associated with opposing depth interpretations can coexist within the model's visual representations. The results suggest that convex- and concave-associated features can be simultaneously present in a limited set of intermediate layers, providing insights into how conflicting visual cues may be represented.

  • Through targeted feature steering, ablation, and other experiments, the paper goes beyond correlational feature analysis to examine causal effects on model outputs.

Critical weaknesses
  • The paper's experimental design and findings are based on a central assumption: the texture and contour cues provide conflicting depth interpretations. However, this raises a foundamental concern that, if one or both cues are ambiguous, the combined stimulus may therefore fail to present a clear and controlled conflict between two opposing depth cues. Simply saying, how can one ensure that the texture and contour themselves are free from significant perceptual ambiguity?

  • The paper lacks the encoder-level validation, which I think is very important. Since the analysis of this paper assumes that the CLIP vision encoder retains the texture and contour information necessary for distinguishing convexity from concavity. However, whether these fine-grained visual details are well-preserved throughout the encoder is unclear. Features such as texture gradients and contour geometry could be easily changed, abandoned before the SAE analysis. Therefore, the authous should evaluate whether the intended depth cues remain decodable from the encoder representations, for example through layerwise probing on the single-cue control stimuli.

  • As acknolewdged in the paper, all findings and experiments are made from one single model. However, VLMs differ substantially in their visual encoders and vision–language integration mechanisms, including projection-based architectures, query-based visual resampling, etc. The different architecture might largely affect how these fine-grained textures and cues preserved, processed, and propagated to the model. This could largely limit the gneralizability of the paper's conclusion.

  • The finding in Section 4.1 has exposed a substantial limitation, which should be further elaborated. The result shows that that the model's responses are significantly influenced by the positions of the options. As the model constantly selects option A in all configurations. Therefore, the baseline responses do not establish a consistent semantic depth judgment. This further triggers my concern to the results in the upcoming sections. In Section 4.5, among the 15 "same-direction" paths, none exhibit a change in the same direction under both A/B mappings. The authors should evaluate whether the identified paths remain valid in a position-invariant scenario.

Questions for authors
  • The example shown in Figure1 is not very convincing to me. For the reason that, the contour seems to be perceptually ambiguous between a concave valley and a convex ridge. This triggers my concern to the first weakness. In other words, how do the authors validate that the contour-only stimulus reliably conveys concavity? The authors need to independently validate the six categories of control stimuli and report the consistency of judgments across these stimuli, rather than relying solely on pre-assigned "convex" or "concave" labels.

  • In Line 160, the authors state ' The exact cross-cue numerical thresholds were not supplied in the available record, limiting independent reconstruction of this screen.' Could the authors clarify what does this mean?

  • Another fundamental question is, how do the authors validate that the selected SAE features genuinely represent convex and concave depth cues, rather than low-level visual patterns or stimulus-specific artifacts? The authors should further validatre the identified SAE features, as this will foundamentally affect all the findings in this paper.

Reviewer_o8q5OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates how VLMs represent conflicting depth cues when determining whether an image is convex or concave. The authors use Mach-card stimuli where texture and contour provide conflicting depth information. They train SAEs on LLaVA-1.6 to identify cue-selective and cross-cue depth features. They then use steering, ablation, and cross-layer contribution replacement to investigate how these features influence the model's predictions. They find that opposing depth features coexist in intermediate layers and that interventions can change the model's convex/concave logit margins. They also find that part of an earlier cue feature's effect is carried through a later depth-direction feature, with a larger effect for the concave path.

Critical strengths
  • I think the research question is interesting. Using conflicting visual cues to investigate how VLMs represent and integrate visual information is a promising direction.

  • I like the combination of SAEs with causal interventions, particularly the attempt to trace how earlier cue-selective features contribute to later depth-direction features.

Critical weaknesses
  • The paper only considers LLaVA-1.6-Vicuna-7B, which is a relatively old and small VLM. I understand that training SAEs is expensive, but it is hard to make claims about how VLMs represent conflicting depth cues when only studying features in one model. How do we know these findings are not specific to LLaVA? Without additional (updated) models, I doubt the generalizability of this paper.

  • I am concerned that it seems the models struggle to even complete this task. What is the model's accuracy on the 150 control images? I would like to see accuracy for both texture identification and contour-based depth estimation. Can the model reliably identify convex and concave geometry from either cue?

  • The authors report that the model selects option A in all four baseline evaluations, even when the answer mappings are swapped. This is very concerning. If the model cannot actually distinguish convex from concave images, how can we meaningfully interpret the SAE features as representing depth? Are these features actually related to depth estimation, or are we just shifting the model's answer preferences?

  • The very small logit differences in Table 2 exacerbate my concerns. In addition to logit margins, I want to see how many answers actually change after interventions. Further, none of the 15 selected cross-layer paths show consistent effects across both answer mappings. I think this substantially weakens the causal claims.

  • The authors only evaluate the T+/C- conflicts. Why not consider the T−/C+ conflict? If the goal is to identify how conflicting depth cues are represented, looking at only one direction of conflict is a major weakness. The paper underexplores how texture and contour contribute to depth estimation. It would be interesting to vary the textures and contours independently, including textures that provide no convex or concave depth information. This would help determine whether the identified features actually represent depth or simply respond to particular visual patterns.

  • The authors only use 25 images per class, totaling 150 images, to identify cue-selective features. This is unnecessarily small. I don’t understand why texture gradients, contour angles, and other properties can’t be adapted to generate thousands of images.

  • How stable are these features across different images? I would like to see whether the identified features generalize to a larger held-out dataset rather than just the specific stimuli used for screening.

Questions for authors
  1. How do we know these findings are not specific to LLaVA?

  2. What is the model's accuracy on the 150 control images?

  3. Can the model reliably identify convex and concave geometry from either cue?

  4. If the model cannot actually distinguish convex from concave images, how can we meaningfully interpret the SAE features as representing depth?

  5. Are these features actually related to depth estimation, or are we just shifting the model's answer preferences?

  6. Why not consider the T−/C+ conflict?

  7. How stable are these features across different images?

Comments on the paper

1 comment

Dear authors, it looks like you haven't responded to reviewer concerns yet. As the rebuttal phase is nearing its end, this is your last chance to respond - alternatively, I would suggest to consider withdrawing the paper if you don't believe it has a chance of acceptance.

Thank you,

Your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.