acceptodds
Under review as a conference paper at ICLR 2027

The Umbra Effect: Use–Elicitation Dissociation in Vision-Language Models

Abstract

Anthropic's J-lens study links verbalizable representations to a global workspace in language models. This raises a complementary question for vision-language models: when visual information influences an answer, can an intervention also elicit that information in words? We identify the Umbra effect, a dissociation between causal use and verbal elicitation of visual information, and characterize it using mirrored-image interchange, linear probes, and calibrated signed activation injection. Across five checkpoints from two model families, we characterize three findings: 1) Use–elicitation dissociation. Four Qwen checkpoints show weak spatial pooled-direction elicitation in tested settings where interchange changes answer margins, despite effective color controls. 2) Vision-to-answer handoff. A detailed Qwen2.5-VL-3B study localizes a crossing of vision- and answer-position interchange effects at L23, within a band comprising six tested layers with causally influential, linearly decodable spatial representations. 3) Payload-dependent elicitation. LLaVA-OneVision responds to position-preserving spatial payloads where pooled payloads fail, identifying a boundary on the cross-model pattern. Photograph and clutter tests additionally extend the use–elicitation comparison within the 3B case study. Together, these results motivate auditing causal contribution, decodability, and controlled elicitation as distinct measurements, with payload and dose calibration guiding the interpretation of negative readouts.

Primary area:
interpretability and explainable AI
Keywords:
vision-language models, mechanistic interpretability, causal interventions, activation steering, linear probing, verbal elicitation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_v5TdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates when visual information influences an answer, can an intervention also elicit that information in words? In exploring this question, it identifies a use-elicitation dissociation (termed the Umbra effect): visual relations influence answers under interchange yet respond weakly to elicitation with the tested payloads. Furthermore, the paper discovers a vision-to-answer handoff and payload-dependent elicitation across five checkpoints from two distinct model families.

Critical strengths

The mathematical formulation in the methodology section is academical.

Critical weaknesses
  1. Does the Umbra effect identified in the paper possess generalizability? The paper presents Qwen2.5-VL-3B as a detailed case study, while other checkpoints are utilized to examine model variations. However, the detailed case analysis is conducted on only a single model, and merely five models are used for the overall evaluation. Consequently, the number of observed models is highly limited.

  2. Regarding Figure 4, it is necessary to present the "Vision and answer positions exchange their relative influence" plots for Qwen2.5-VL-7B, Qwen2-VL-2B/7B, LLaVA-OneVision-7B, and additional models, in order to uncover universally existing patterns.

  3. The rationale behind selecting the 25 specific layers out of the total 36 layers is not elaborated.

Questions for authors

Do the Umbra effect and the vision-to-answer handoff identified in the paper possess generalizability, rather than being confined to a single model? For specific details, please refer to the "Critical Weaknesses" section.

Reviewer_kBjKOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2(+1)
Presentation
1
Soundness
1
Read review
Paper summary

The paper tests whether visual information a VLM can also be elicited in words. On mirrored image pairs (left/right, above/below), it compares three measurements: causal use (swapping vision activations), linear decodability using probes, and elicitation (injecting a relation direction into a neutral prompt). In four Qwen models, swapping vision activations changes answers, but injecting a pooled spatial direction elicits nothing. The authors call this the Umbra effect.

Critical strengths

The figures are legible. Although they are very hard to comprehend.

Critical weaknesses

The paper has serious problems with scientific writing and clarity. Key terms are used without being introduced or defined; for example, it is unclear what “mirrored partner” means. The motivation for the research question is unclear. Although individual sentences are grammatically correct, they do not form a coherent explanation, making the paper’s arguments difficult to follow.

Questions for authors

What does the term 'payload' refer to in this paper?

What do the authors mean by effective color controls?

Reviewer_T2LWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the dissociation between causal use and verbal elicitation of visual information in vision-language models. Using mirrored image pairs, it combines activation interchange (causal influence on answers), linear probes (decodability), and calibrated signed activation injection (elicitation of relation words), with color injection as a positive control. There are 3 main findings: across four Qwen checkpoints, spatial relations show zero pooled-injection success at layers where interchange changes answers, while color injection fully succeeds; on Qwen2.5-VL-3B, an Umbra band spans L17–L24, and vision-position causal influence is overtaken by answer-window positions at L23; LLaVA-OneVision responds to position-preserving payloads (0.58–0.83 success) where pooled payloads fail, showing elicitation depends on payload form. Validation includes permutation tests with Bonferroni correction, bootstrap intervals, multi-seed repeats, threshold-sensitivity analysis, semantic/counting tasks, COCO and cluttered-scene transfer, and a J-lens check showing the readout tool itself fails to recover known facts. Limitations: small samples, only five checkpoints from two model families, and oracle repair requiring supplied labels.

Critical strengths

I like these findings and insights in this paper based on the comprehensive experiments.

  • The dissociation is established as a measurement fact rather than a single-method artifact. Interchange, probing, and injection are compared on matched layers and stimuli, and color controls at the same site, layer, and dose succeed at rate 1.00 while spatial pooled injection is exactly 0 — ruling out a broken injection pathway. Documenting this kind of measurement disagreement directly changes how negative steering or probing results should be interpreted.

  • Experimental hygiene is unusually careful: dose is calibrated on color so spatial outcomes never enter calibration; permutation tests survive Bonferroni correction across 25 layers; bootstrap CIs, control- and stimulus-seed repeats, and pre-declared threshold sensitivity (UI of .80/.23/.05 at probe cutoffs .45/.50/.55) are all reported, even where they weaken the headline. The J-lens validation failure is a useful instrument-level cautionary result on its own.

  • The LLaVA position-preserving result turns a null into a boundary condition: structured payloads succeed (0.58–0.83) exactly where pooled payloads fail, showing the elicitation gap tracks representational format rather than absence of information — a concrete demonstration of why payload and dose must be made explicit in elicitation audits.

Critical weaknesses
  • The Umbra band and Umbra Index depend strongly on fixed thresholds. Moreover, the reported UI interval holds layer membership fixed and does not propagate uncertainty from probe accuracy, elicitation rates, or threshold-based band assignment. The stability of the reported band boundaries and UI values is therefore unclear.

  • The color control shows that activation injection can affect generation, but it does not show that the same intervention can reliably elicit a binary spatial relation. Notably, the token-derived relation control fails at five of the six band layers. Thus, zero spatial elicitation cannot yet be clearly separated from a relation-specific mismatch between the payload, injection site, and recipient context.

  • The main results rely on small and selected stimulus sets: the causal analysis uses 23 baseline-correct pairs, key injection experiments use only 6-16 pairs, and the COCO experiment contains nine valid pairs. For constant outcomes, bootstrap intervals such as [0,0] and [1,1] describe the observed sample but do not quantify uncertainty on new stimuli.

  • The claimed vision-to-answer handoff at L23 is based on a descriptive crossing between estimates computed from different sample sizes: 23 pairs for vision positions and 12 for answer positions. The paper does not directly test their difference on the same examples, and the sparse layer grid does not establish that the crossing occurs specifically at L23.

Questions for authors
  • Can the authors recompute the Umbra Index using a resampling procedure that refits the probe and recalculates elicitation-based layer membership? How stable are the band boundaries and UI across resamples and reasonable choices of all three thresholds?

  • Given that the token-derived relation control fails at five of the six band layers, what evidence shows that these layers can support relation-word elicitation under the current setup? Could the authors add a successful relation-matched positive control or test a recipient context containing identifiable objects?

  • Do the all-zero and all-one injection results replicate on independently generated stimulus sets? Could the authors report exact or Wilson intervals for the raw success rates and show results both with and without conditioning on baseline-correct pairs?

  • Is the L23 handoff supported by a paired comparison of vision-position and answer-position effects on the same examples? Please report uncertainty for their difference and evaluate the missing intermediate layers if possible.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.