acceptodds
Withdrawn from ICLR 2027

Does the Workspace See? Testing and Inducing a Multimodal Global Workspace in Vision‑Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

The Jacobian lens identifies a small, workspace-like subset of internal representations through their average influence on model outputs. A readout indexed by the output vocabulary can only return tokens, so for vision-language models we cannot tell whether visual content lies inside the subset or outside it. We call the question of whether a representation can be expressed in the symbols of a given output channel channel expressibility. Verbalizability, the property the lens reports for the subset, is the special case where that channel is text; a single-channel model cannot test it, since its readout has only that alphabet. We introduce MJ-lens, a modality-split Jacobian lens (four operators in a unified model, two in an understanding-only model) that also adds a readout into the discrete visual codebook by linearizing the image-generation head. We test whether the workspace account extends to multimodal models, using understanding-only and unified models. We find that the subspace MJ-lens selects has a causal role and its own transport geometry, but it does not satisfy all workspace signatures. In the understanding-only model, ablating this subspace impairs some visual judgments, but nameability does not predict which ones. In the unified model, the output channel changes what the subspace contains, yet overwriting a lens coordinate does not change what the model draws. Directions obtained without the lens do change it, installing a specified object only at early layers and a colour across a broad middle band, while the lens’s own coordinates move nothing. The bound is on the instrument, not the representation: the lens can read the subspace, but as a readout operator it does not provide directions that control what the model draws.

Primary area:
interpretability and explainable AI
Keywords:
mechanistic interpretability, vision-language models, global workspace, Jacobian lens, multimodal representations, verbalizability, logit lens

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_KMX6OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper aims to expand the J-lens into multimodal models. They propose the MJ-lens, which aims to perform J-lens like early layer readout on activations on image and text latents for Qwen-VL, and image-text, text-image, etc. channels for Janus Pro.

Critical strengths
  • The idea of applying a J-lens like readout to models which output both text and images is interesting.

  • The authors explore methods to steer model behavior with their multimodal lens.

Critical weaknesses
  • The title asks "Does the Workspace See?", but the paper contains very little (weak and noisy) information about whether the "workspace" induced by J-lens is actually visual. Of course we can apply a visual unembedding module to the activation if one exists for the model. But the authors do not creatively extend these multimodal unembedding modules to explore whether the latent space is "thinking" visually.

  • The main finding in this paper goes against the original J-lens finding that steering to alter the J-lens readout changes the model behavior. The authors actually find that in Qwen-VL and Janus Pro, the two model families they study, this is not the case. But why would this be? Is there a control on LLM-only models which show that steering through the J-lens does change outputs in LLMs, so the finding is unique to multimodal models? Why is difference-of-means not necessary for Claude?

  • The writing of this manuscript is very obscure. For example, this sentence that closes the abstract: "The bound is on the instrument, not the representation: the lens can read the subspace, but as a readout operator it does not provide directions that control what the model draws.” --> it could have just been "We find our MJ-lens can perform readouts but does not extend to steering".

Minor Weaknesses:

  • The idea of applying a logit-lens like readout on multimodal activations is not entirely new. It has been performed previously in [1], for example. The Jacobian Lens is just a small modification on the logit lens setup. I say this is minor because this manuscript also investigates Janus Pro, but I still find the contributions to be quite narrow.

References: [1] https://arxiv.org/pdf/2410.07149

Questions for authors
  • If the MJ-lens does not enable steering - what might we use it for? Is it a reliable monitoring tool for misalignment? How does adding the extra modality expand what abilities we would have had with just the text monitoring? Why should we be interested in the MJ-lens?
Reviewer_484EOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether the global workspace hypothesis and the Jacobian Lens can be extended to multimodal models. The authors introduce MJ-lens, a modality-split extension of the Jacobian Lens, and analyze visual representations through probing, ablation, and activation interventions.

Critical strengths

The paper explores an interesting research question concerning the interpretability of multimodal representations and the relationship between information readout and causal control.

Critical weaknesses

1.Limited methodological novelty. The proposed MJ-lens primarily extends the existing Jacobian Lens by separating modality-specific operators and introducing a visual readout. The paper would benefit from a clearer explanation of what fundamentally new capabilities the proposed framework provides beyond existing interpretability methods.

2.Insufficient generalization across multimodal architectures. The image-generation intervention experiments primarily focus on Janus-Pro. It is unclear whether the observed limitations of MJ-lens generalize to other unified multimodal models with different architectures and generation mechanisms. Additional experiments would substantially strengthen the conclusions.

Questions for authors

The proposed approach feels somewhat lacking in innovation; if the authors could provide additional experiments or tests in other scenarios, I might consider raising the score.

Reviewer_RiQSOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper asks whether the text-only Jacobian lens misses visual information in a model's internal states. The authors introduce MJ-lens to study effects between visual and text positions, and add an image-codebook readout for Janus-Pro. Removing selected states changes some visual judgments, but whether the states can be named does not predict which tasks are affected. Lens directions do not effectively change generated images, while mean-difference directions do. A direction that reveals information may therefore be poor at controlling behavior.

Critical strengths
  1. Readout and control: The authors test what the lens can read and what its directions can change. In the same model, mean-difference directions change generated objects and attributes while lens directions barely do, giving the negative result a useful positive control.
  2. Readout checks: The authors report where readouts match final outputs and find that image readouts at visual-token positions mostly echo the input. They avoid treating every visualization as newly formed visual content.
Critical weaknesses
  1. The experiments do not answer the workspace question. They show that some internal states affect visual tasks and that lens directions fail to control image generation. They do not show whether a functional global workspace exists, so the conclusion should focus on what the lens can and cannot do.
  2. The nameability claim rests on limited evidence. The study tests five synthetic stimulus types without fully separating nameability from visual attributes and question format. The reported p=0.81p=0.81 does not show that the groups have equal effects.
  3. The output-modality test changes too many things. The text- and image-output conditions use different inputs and prompts, so output modality alone may not explain the difference in visual components. Image-generation edits are tested mainly in Janus-Pro, so it is unclear whether the result holds in other models.
Questions for authors

See above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.