acceptodds
Withdrawn from ICLR 2027

Pointer–Payload–Handoff: Tracing How Visual Evidence Reaches the Answer in Vision–Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

A vision-language model can attend to the right region and still answer incorrectly. Spatial grounding identifies where a model looks, not how the selected fact reaches the computation that chooses the answer. In Qwen3.5, the strongest spatial pointer even appears after the image-to-question handoff, and transplanting the grounded attention contribution itself fails to repair answers. We trace one answer-determining visual fact by asking four questions: where it is first causally available, which computation moves it downstream, where the transferred content is carried, and through which state it reaches the output. Pointer–Payload–Handoff (PPH) answers these questions with three complementary interventions—restoration, blocking, and semantic interchange—at coordinates frozen before held-out confirmation. Independent analyses in Qwen3.5, Qwen3-VL, and Gemma 3 recover the same ordered role graph, S → W → C → R, at architecture-specific depths. Writers range from one dominant attention component to distributed modules, yet every family certifies the directed links S → W, S → C, and C → R. The W → C adjacency holds in Qwen3.5-9B, where it is measurable. The frozen carrier is causally usable: replacing it with a state built from correct visual evidence repairs 19–35% of initially incorrect real-image answers across the three families, against near-zero matched controls. With the task-relevant option text present, the carrier mediates 58–89% of the source effect across the three families. Replacing that block with equal-length unrelated text reduces mediation by 54 points in each family; instructing the model to ignore the options leaves the point estimate close to the original. These controls isolate the effect of the option text, not the answer word within it. Interpreting a vision-language model therefore requires stating the prompt condition under which its mechanism was recovered.

Primary area:
interpretability and explainable AI
Keywords:
Vision-Language Models, Mechanistic Interpretability, Causal Provenance, Spatial Attention, Post-Training

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_e1FQOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The authors are trying to understand at what point information in the network of an LLM effects its answer by replacing internal states taken during answering with states from a model processing the same image. They find that copying hidden states associated that are associated with the question text can correct errors even when copying information from the strongly attended image region does not help. They test across several different modifications of the original image, such as color, and against random replacements.

Critical strengths
  • Compared to other prior work I have seen, they provide more ablations (such as the text length, multiple choice, different types of transfer) on which specific states contain the correct information and can causally effect the answer.
Critical weaknesses
  1. The writing is unclear and extremely difficult to follow, so much so that is obscures the point of the paper
    • The abstract is confusing and non-specific; it doesn't define terms such as "spatial grounding" (L13), spatial pointer (L16), and the graph and its parameters of S, W, C, R (L22), etc. It is very difficult to understand the contribution here and this must be entirely rewritten.
    • The introduction writing is disjointed and sounds very LLM-like; many details are introduced and not explained, but they are vital to understanding it's flow. This in itself is not a disqualifier, but I find it very difficult to understand ("can look at... and still give the wrong answer" (L39); "Spatial attention, probing... establish where a fact is. A causal... changes the output." (L45-46, "the carrier’s content tracks the donor. " (L93); "Restoration establishes that a state.. and semantic interchange establishes which content a state actually carries" (L85)). Jargon is made up and not explained well )(e.g.,"donor", "carrier" are barely defined on L52-78).
    • Nonsense sentences in the main body (e.g., "Attention maps and activation patches each expose one facet of a VLM’s visual computation, but neither certifies the path this paper follows" (L148): what do you mean by certify the path that this paper follows?)
    • The statements in the experimental setup are not well defined and are also opaque. E.g: "A deterministic 200-pair diagnostic set changes one answer-determining visual fact while preserving the remaining layout and prompt" (L254)- I do not know what this means
    • I found it incredibly difficult to understand what the experiments were actually testing
  • The analysis lacks substantial insight
    • For instance, what is the takeaway? Salazar et al. (Pathways of Visual Information Flow in Vision-Language Models) already have similar findings about...
    • While the paper does appear to have more controls than prior work, I don't see them motivated or unified anywhere (e.g., they test the changing the answer format. Was there reason to believe this would be significantly different? or which attributes transfer well. They find that color does not(L464–466) but do not adequately explore why this is)
    • In general, the paper appears to be more of an assortment of semi-related results than a unified narrative.
Questions for authors
  • Are the photos actually matched, and if so, how do you source these photos with only a single difference? (Are they edited?) Or are they just extremely similar photos, but the one attribute is not the only difference?
  • There is a lot of related work you did not contextualize or mention, such as "The Narrow Gate: Localized Image-Text Communication in Native Multimodal Models" (serra et al, 2025) and "Linear Mechanisms for Spatiotemporal Reasoning in Vision Language Models" (Kang et al) that seem to do materially similar things to your work- what is the difference?
  • Can you describe in more detail how you transplant the states? Is it the activations, and at what point? I assume it is the activation stream after some layer, but I think it would be helpful if this was defined precisely.
  • What exactly is Figure 1 trying to show? On the left, is the spatial pointer still referring to the same image as the mug with the cat? What are the dotted diagrams in the middle in a, b and c? Do they represent token positions?
Reviewer_FMLoOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper looks at the question of attention processing in vision language models. It is motivated by the phenomena that correct attention masks may still produce incorrect visual question answers. The pipeline looks to make use of "counterfactual" images that change object properties (e.g., making a red mug blue), and changes in answer logits on the modified images are used to "interpret" where and how answers are mechanistically produced in the original model. The authors use their approach and custom dataset to analyze well-known VLMs such as Qwen and Gemma. It seems as if experiments are presented on 200 custom images, but this is not clearly described. I also don't see how counterfactuals are generated - perhaps with a image-editing generative model?

Critical strengths
  • The core goal of analyzing the mechanistic behaviour of visual question answering models via controlled exposure image inputs is reasonable. But this is written in such an opaque manner that I would strongly argue for its rejection; our community needs to do better.
Critical weaknesses
  • The salient aspect of this paper is that it is nearly impossible to understand, and looks to be heavily AI generated. I will challenge anyone to read the abstract and describe what this paper is about. IMHO (the AC or other reviewers are free to disagree), this paper should have been desk rejected.

  • Core technical aspects are mysterious. What is the definition of a "repair"? I also do not know what is being evaluated in the tables; I would have thought it would be correctness of multiple-choice-questions, but it appears to be some kind of mechanistic analysis.

Questions for authors

This work needs a substantial rewrite. I believe these kinds of submissions hurt the field since they take resources away from efforts that genuinely try to add to the scientific discourse.

Reviewer_pcCqOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Pointer–Payload–Handoff, a protocol for tracing how located visual evidence influences a vision-language model’s Q&A. It combines activation restoration, downstream blocking, and semantic interchange to identify source, writer, carrier, and readout roles. Experiments across Qwen3.5, Qwen3-VL, and Gemma 3 recover related functional mechanisms at different depths. The paper further shows that task-relevant option text substantially changes the contribution of question-side carriers.

Critical strengths

The paper has several strengths. First, the distinction between spatial attention and causal use of visual evidence is well motivated, and the protocol separates efficacy, routing, and semantic content rather than treating activation restoration alone as a complete explanation. Second, independent discovery, validation, and held-out confirmation provide a comparatively careful experimental design. Third, the prompt controls are informative: replacing options with unrelated text of equal length would largely reduce carrier mediation, whereas asking the model to ignore the retained options has much less effect.

Critical weaknesses
  • Scale and pathway validation: The main experiments use 4B models, and the writer-to-carrier link is separately confirmed only in Qwen3.5-9B. More comprehensive 9B results and larger-model replication would strengthen the serial-pathway claim.
  • Cross-model comparability: How are equivalent roles identified despite different search spaces and component structures? Since all tested models are prefix-fusion decoders, the architectural requirements and broader applicability of PPH remain unclear.
  • Prompt-dependent routing: Reduced mediation at MCQ-selected coordinates may reflect carrier relocation rather than weaker question-side processing. Independent role discovery under no-option prompts would strengthen.
  • Practical implications or intervention impact: What concrete recommendations follow for improving VLM grounding and evidence use? The paper should identify which components to target and distinguish tested improvements from hypotheses.
Questions for authors
  • If roles are independently discovered and confirmed under the no-option prompt, does question-side mediation remain weak, or does a different carrier emerge?
  • What additional intervention would distinguish a serial source–writer–carrier pathway from parallel source-to-writer and source-to-carrier pathways in the 4B models?
  • Why does color fail semantic interchange despite passing the causal-role tests, and does this failure persist with donors matched more closely in object identity and context?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.