acceptodds
Under review as a conference paper at ICLR 2027

Question Tokens Relay an Answer-Level Summary in Vision-Language Models

Abstract

Activation patching asks how vision-language models (VLMs) use visual information by copying hidden states from a donor input into a recipient input during inference. However, changing the donor image usually changes both the image content and the correct answer, confounding the two. To disentangle these factors, we introduce a controlled activation-patching setup that holds the question fixed while varying whether the donor shares the recipient's answer. We construct 538 quartets from VQAv2, each pairing one question with four images, two supporting each of two answers, e.g., two with ”yes” and two with ”no”. Within each quartet, we patch the recipient with question-token states from either a same-answer or different-answer donor at a given layer, while matching per-token norms. In all seven checkpoints we study, different-answer donors pull the recipient toward the donor's answer; in six, same-answer donors change the full output distribution far less. Yet same-answer donors are not neutral: in four checkpoints, a recipient moves one-third to two-thirds of the way toward their donors' own answer margins. Later layers thus read an answer-level summary from the question tokens, namely which answer the image supports and how strongly, and a layer sweep in three models places this summary in a narrow band of middle layers. In each of the three VLMs, a learned rank-16 component preserves the donor effect when retained and suppresses it when removed, more strongly than magnitude-matched controls. The donor effect is weakest when the intervention layer lies beyond this window, so patching studies should control both the donor's answer and the layer.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Vision-Language Models, Activation Patching, Multimodal Representations, Mechanistic Interpretability

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_n3VnOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents a mechanistic analysis on the information flow of hidden representations of question tokens and how this information affects the model's prediction. The authors use activation patching as the main approach to compre the effects of replacing question token hidden states with those from different images (sharing or differing in answers) to the same question. The experiment spans multiple multimodal LLMs by constructing 538 quartets from VQAv2 and found that different-answer donors likely to drive the recipient toward the donor's answer whereas same-answer donors have less impact on output. Additionally, the authors found that the question-token representations can be decomposed into a low-rank subspace that captures part of the patching effect. Authors demonstrate that maniuplating this subspace representation can also have causal impact on model's prediction.

Critical strengths
  1. The experimental design in this paper is sensible. Though I have some reservation of the problem scope (see weakness), I believe the experiments presented in this paper reasonably support the conclusion. Specifically, the paper intoduces a controlled activation patching setup while keeping the question fixed and matching the hidden norms, which helps disentangle the effects of the answer-related information (the main study subject in this paper) and other variations.

  2. In addition, the experiment covers a wide range of multimodal architectures and different protocols such as layer-wise patching, low-rank subspace interventions, which make the experiment more compact.

Critical weaknesses
  1. The problem scope of this paper and research insight is incremental to me. The main conclusion of this paper can be summurized as, the hidden states of question tokens can carry answer-related information in the middle layers and subsequent layers in the LLM use this information to make prediction. Although this exact conclusion is not stated by existing work in literature, a number of existing work studies the information flow of MLLM and some of them even studied this in a more generalized setup [1, 2, 3]. [1] and [3] already studied the information flow within MLLM and [3] even shares some conclusion of this paper that the visual information can be mediated through question tokens to affect final predictions of MLLMs. The results related to middle layer integration are also presented by existing work [2].

  2. The conclusion in this paper is well-expected given the current literature in community. It is not clear to me what additional insights I learned from this paper. In my current understanding, this study mainly focuses on the question token hidden states. But what additional understanding do we obtain by speficially studying the information flow of question tokens? I don't think there's enough justification on why question tokens are an important subject study or what broader impact this observation can make (e.g, improving model performance).

  3. Even for this specific problem of information flow through question tokens, I think the study is not significant enough to publish at top-tier conferences. For example, the paper conducts a layer-wise study on patch layer yet the patch layer in other experiments do not make use of the result. Specifically, the layer sweep shows a peak of patching at 13 and 14 for Gemma3 and Idefics3 but the patch layer in other experiments such as low-rank subspace are still other predefined patch layers. Additionally, the experiment results show that different MLLM have different patching effect yet the paper doesn't investigate much into these aspects.

  4. This is a minor point but the writing in this paper can be improved. While individual sections are written clearly, the transition and motivation of different studies are very weak. In the current form, it read like "We test A and A holds; we test B and B holds" but why these studies matters is not clearly justified in the paper. This is also related to the very motivation of paper: why studying question tokens is important?

[1] Basu et al, Understanding Information Storage and Transfer in Multi-modal Large Language Models [2] Yu et al, How Multimodal LLMs Solve Image Tasks: A Lens on Visual Grounding, Task Reasoning, and Answer Decoding [3] Salazar et al, Pathways of Visual Information Flow in Vision-Language Models

Questions for authors
  1. Given the existing literature, could authors clarify on the significance of this study in terms of both the new insights this study establishes and the potential broader impact?

  2. More related to the specific problem, could authors clarify why the information flow of question tokens is an important study subject?

  3. As discussed in weakness, the results of the layer sweep are not made use in other studies such as the investigation of the low-rank subspaces. How would the results for low-rank subspace change if we use the peak-effect layers to learn the projection? Or more broadly, would there be a more holistc landscape for the low-rank subspace across different layers?

  4. Could the authors also clarify how the pre-specified intervention layers were originally selected?

  5. Do authors have any insights or hypothesis regarding the difference of patching effects across different MLLM families? Ideally, it would be helpful if these hypothesis can be backed by empirical evidence.

6 comments
AuthorsOpenReview ↗

Response to Reviewer n3Vn (1/3)

Dear Reviewer n3Vn,

Thank you for your detailed review, for finding our experimental design sensible and our experiments reasonably supportive of our conclusions, and for noting that the fixed-question, norm-matched setup disentangles answer-related information across a wide range of architectures and intervention protocols. In the revised PDF, we mark changes addressing your review in blue and changes addressing concerns shared with other reviewers in purple. We refer to your weaknesses as W1–W4 and your questions as Q1–Q5, in the order of your review, summarize each point before our response, and tag each change in the PDF with the same numbers.


W1. The problem scope and research insight are incremental, because existing work already studies information flow in MLLMs, including visual information mediated by question tokens and middle-layer integration.
W2. The conclusion is well expected, and the paper does not justify its additional insight, why question tokens are an important subject, or its broader impact, such as improving model performance.
Q1. Clarify the significance of the study, in terms of both the new insights it establishes and its potential broader impact.

A1: Thank you for pointing us to these works. Our main point is that prior work shows where visual information travels, whereas our design reveals what the question tokens relay. Basu et al., Yu et al., and Salazar et al. trace storage and transfer, processing stages, and pathways, but none of them holds the answer fixed while changing the image. Our same-answer donor adds this control, and it yields four findings that these designs cannot establish:

  • The summary is graded. A recipient inherits one-third to two-thirds of a same-answer donor's answer strength in four checkpoints, as Section 4 of the original submission reports.
  • Content effects can look like answer effects. In early layers, both kinds of swap change the output about equally, as Figure 3b shows, so designs without a same-answer control would credit content effects to the answer.
  • The component follows the answer, not the letter. We newly show that when donor and recipient prompts assign opposite letters to the same answers, the component moves the recipient toward the donor's answer by 1.67, whereas copying the donor's letter would predict the opposite direction. We added this as a new paragraph in Section 5.4, with details in the new Appendix H.6.
  • The component generalizes. We newly show that the component meets all six original criteria on the 237 pairs whose answers were never used for fitting. We added this to Section 5.2, with details in the new Appendix H.4.

These findings also have broader impact: the component can control generated answers, the best intervention layer depends on the type of intervention, and patching studies of VLMs need a same-answer control. We added these points as a new "Implications" paragraph in Section 6. We also added Yu et al. to the introduction and to the related work in Section 2, and revised both to state this distinction.

Therefore, our study establishes what the question-token pathway carries and how the model uses it, which the routing analyses of prior work leave open.


Q2: Clarify why the information flow of question tokens is an important subject of study.

A2: Thank you for this question, which also covers the question-token part of W2. We study the question tokens for two reasons, and we added both to the first and third paragraphs of the introduction. First, later layers rely on these states: swapping only the question-token states makes 78% of the answers of Qwen2.5-VL-7B follow the donor, even though the recipient keeps its own image tokens, as Section 5.4 of the original submission reports. These states are therefore where image evidence becomes a decision. Second, their text is identical across images, so their states differ only in what they read from the image, which makes them a controlled site for our comparison.

We also tested whether the answer contrast resides in these tokens, by applying the same swaps to the image tokens and the final prompt token. In Qwen2.5-VL-7B at layer 17, the contrast is 1.22 for question tokens, but only 0.004 for image tokens and essentially zero for the final prompt token. In Idefics3 at layer 14, image tokens carry a contrast as large as question tokens, which marks a genuine difference in how the two models route the answer. We added this test as a new paragraph in Section 4, "Which token positions carry the contrast?", with details in the new Appendix H.1.

Therefore, the question tokens are both the cleanest site for an answer-controlled test and, in Qwen2.5-VL-7B, the site where the answer contrast is concentrated.

AuthorsOpenReview ↗

Response to Reviewer n3Vn (2/3)

W3. The layer-sweep results are not used in the other studies: the sweep peaks at layers 13 and 14 for Gemma3 and Idefics3, yet the low-rank analyses use other, predefined layers. The paper also does not investigate why patching effects differ across MLLMs.
Q3. How would the low-rank results change if the projection were learned at the peak-effect layers, and is there a more holistic landscape across layers?

A3: Thank you for this suggestion. We find that the component becomes much stronger when fitted inside the window, and that its different uses peak at different depths. We added these results as a new Section 5.5, with details in the new Appendix H.5:

  • Idefics3. We refit the component at the sweep-selected layer 14 on the same 320 evaluation pairs. The share of the donor effect that it carries rises from 51% to 92%, and its retention and removal specificity rise from 0.28 to 1.14 and from 0.36 to 1.52. The result holds across three initialization seeds.
  • Gemma3. On a separate panel of 270 mostly yes/no quartets, the projected-source effect is 0.20 at layer 13 but only 0.015 at layer 21.
  • Qwen2.5-VL-7B across layers. Retention specificity is 0.01, 0.48, and 0.47 at layers 13, 17, and 21, whereas the shared direction flips 91.6% of yes/no decisions at layer 17 but only 1.6% at layer 21.

We also tested the sweep-selected layers on new data. On 1,000 new GQA quartets, which we used neither to select the interfaces nor to locate the windows, the contrast at the sweep-selected layer is 5.3 times that at the fixed layer in Idefics3 and about 20 times in Gemma3. We added this to the paragraph "The summary resides in a middle-layer window" in Section 4.

Therefore, the component is much stronger inside the window, and answer transfer and steering should be placed at different depths.


Q4: Clarify how the pre-specified intervention layers were originally selected.

A4: Thank you for this question. We selected the Qwen2.5-VL-7B layer in an earlier pilot and transferred its relative depth to the other models before evaluating them. We added this history to the paragraph "Patching question-token states" in Section 3 and as a new paragraph in Appendix A.3:

  • Qwen2.5-VL-7B. The pilot used 120 GQA examples and screened eight depths. It scored each depth by how well clean question-token and visual-token states restore an image-corrupted answer in all three task families, and it selected layer 17.
  • Other models. Qwen2.5-VL-3B used the same candidate position, layer 22. We mapped the other five checkpoints to the same normalized depth, which gives layers 20, 21, 22, 22, and 20 for Idefics3, Gemma3, Qwen3-VL-8B, InternVL3.5-8B, and Phi-3.5-vision. We fixed these layers before evaluating each model.

This history also supports one of our recommendations. The windows lie at about 0.63, 0.45, and 0.39 of the network depth in Qwen2.5-VL-7B, Idefics3, and Gemma3, so a relative depth transferred from one model can miss the window of another. We therefore recommend sweeping layers before any fixed-layer analysis.


Q5: Provide insights or hypotheses, ideally backed by empirical evidence, about why patching effects differ across MLLM families.

A5: Thank you for this question, which also covers the second part of W3. We find that model differences come from two sources: where the window lies and how each model routes the answer. We added this as a new paragraph, "Why models differ", in the new Discussion section, Section 6:

  • Window depth. Moving into the window raises the share that the Idefics3 component carries from 51% to 92%, raises the projected-source effect of Gemma3 about fourteenfold, and enlarges the GQA contrast of both models, as described under Q3.
  • Routing. Inside the window, Qwen2.5-VL-7B concentrates the answer contrast in its question tokens, whereas Idefics3 carries as much in its image tokens, as described under Q2.

Therefore, models differ mainly in where and through which tokens they carry the answer summary, not in whether they carry it.

AuthorsOpenReview ↗

Response to Reviewer n3Vn (3/3)

W4: The writing can be improved, and why studying question tokens matters is not justified.

A6: Thank you for this comment. We revised the paper so that each study follows from the previous one. Specifically, we made three changes:

  • We added two sentences to the first paragraph of the introduction that explain why the central question matters: if later layers read only an answer-level summary, the question tokens are where image evidence becomes a decision, and the answer decides what patching these states measures.
  • We restructured the rest of the introduction as four questions, each motivated by what the previous one leaves open: whether a swap acts through the answer, where the summary resides, which part of the state carries it, and what it encodes.
  • We rewrote the opening and "Summary" paragraphs of Section 4 to state the competing accounts each analysis tests and what remains open for Section 5, and we added a Discussion section, Section 6, that states what the results imply.

Therefore, each experiment now answers a question raised by the previous one instead of standing alone.


Thank you again for your feedback. We hope our responses address your concerns, and we would be grateful if you would consider updating your evaluation.

Best, Authors

Reviewer_HMm7OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes a controlled framework for activation patching, which keeps the question fixed while changing whether the donor shares the recipient's answer. This paper conducts experiments on diverse models with created quartets from VQAv2 and performs a comprehensive analysis. Experimental results suggest that the question tokens carry the answer-level summary.

Critical strengths
  • This paper proposes a controlled framework that keeps the question fixed while changing images with the same or different answers. This design is insightful.
  • This paper performs extensive experiments and analysis on diverse models.
  • The paper is well-written with clear motivation and presentation.
Critical weaknesses
  • Even images with the same answer can introduce other different factors, while images with different answers may look similar.

This paper introduces images with the same or different answers as donor images, which is interesting. However, even images with the same answer can introduce other different factors, such as object size. In line 300, this paper claims that

Other image content matters much less than the answer.

But as also discussed in line 302, Gemma3 does not show the trend. This raises the concern of whether the current design of choosing image with same answer is proper enough.

One possible setting to reduce the influence of other image content might be: instead of selecting images with the same answer, diffusion models can be used to modify the area of interest in the image, for example, change the object with different appearances, with the answer either changed or not.

  • Different models or question types demonstrate different behaviors.

This paper conducts experiments on diverse models and question types, which is comprehensive. However, as discussed in the paper, not all models or question types show the same trend. I wonder if there are further insights into why this happens.

Furthermore, how about the trend of different tasks, for example, when identifying object status or spatial reasoning questions?

  • How to improve the models based on these findings.

I understand this paper presents a comprehensive analysis of answer-controlled patching. The question is: how could we further improve the models based on these findings?

Questions for authors

Please refer to the weaknesses.

4 comments
AuthorsOpenReview ↗

Response to Reviewer HMm7 (1/2)

Dear Reviewer HMm7,

Thank you for recognizing our controlled framework as insightful, our experiments and analyses across diverse models as extensive, and our paper as well-written with clear motivation and presentation. In the revised PDF, changes addressing your review are marked in orange and changes addressing concerns shared with other reviewers in purple; each is tagged with the point of your review it addresses. We number your three points W1-W3 in the order of your review.


W1: Even images with the same answer can introduce other factors, such as object size; would local edits be a better control?

A1: Thank you for this suggestion. Our new analyses show that measured visual differences, including object size, do not explain the contrast between same-answer and different-answer swaps. We added these analyses as a new paragraph in Section 4, "Measured visual differences do not explain the contrast", with details in the new Appendix H.2:

  • Image similarity. We adjusted the contrast for differences in image similarity, measured with RGB histograms, DINOv2, and CLIP, and in the number of annotated objects. In all seven checkpoints, the contrast barely changes; for example, it moves from 1.255 to 1.234 in Qwen2.5-VL-7B.
  • Object size. We additionally adjusted for the annotated area of the queried object category, which is available for 846 of the 2,152 recipients. All seven adjusted contrasts remain positive.
  • Controlled scenes. Because none of our 39 attempts at locally editing natural images passed quality checks, we rendered 512 synthetic scenes in which the question-relevant factor changes independently of an irrelevant one. In four of the five model–layer settings, different-answer swaps weaken the model's response to the question-relevant factor more than same-answer swaps do.

Regarding the Gemma3 exception you noted at line 302, we find that it reflects the layer at which Gemma3 was evaluated. Inside its window, Gemma3 shows the same pattern as the other models, as Figure 3b shows, and on GQA its contrast at layer 13 is about 20 times that at layer 21. We revised the sentence at line 300 to "Other image content moves the output less than the answer", which states exactly what the KL analysis shows.

Therefore, the contrast reflects whether the donor shares the recipient's answer, not how similar the two images look.


W2: Different models or question types demonstrate different behaviors; what about object status or spatial reasoning?

A2: Thank you for the question. We find that the effect holds across question types, including spatial reasoning, and that most differences across models come from the layer at which each model was evaluated. To test other question types, we built 1,000 new GQA quartets and added the results as a new paragraph in Section 4, "The contrast extends to Other questions and to GQA", with details in the new Appendix H.3. The contrast is reliably positive in all five model–layer settings, and on the 342 spatial-relation quartets alone it is reliably positive in four of them:

Model, layerAll 1,000 quartets342 spatial quartets
Qwen2.5-VL-7B, 172.182.37
Idefics3, 141.251.16
Idefics3, 200.240.33
Gemma3, 130.980.64
Gemma3, 210.050.01

Object-status questions such as "Is the television turned on?" are part of our original VQAv2 panel.

The only setting without a reliable spatial effect, Gemma3 at layer 21, lies past Gemma3's window. This follows the broader pattern: when we refit the component inside each model's window, Idefics3's share of the donor effect rises from 51% to 92% and Gemma3's effects grow 14 to 20 times,as we report in Sections 4 and 5.5. Beyond the layer, we also find a genuine routing difference: inside the window, Qwen2.5-VL-7B carries the answer mainly in its question tokens, whereas Idefics3 also carries it in its image tokens. We added this as a new paragraph, "Why models differ", in the new Discussion section, Section 6.

Therefore, models differ mainly in where they carry the answer summary, not whether they do.

AuthorsOpenReview ↗

Response to Reviewer HMm7 (2/2)

W3: How to improve the models based on these findings.

A3: Thank you for the question. Our findings suggest three ways to improve VLMs, because they identify where a model forms its answer and how that answer can be changed:

  • Editing answers. We show that the rank-16 component controls what the model says. In Qwen2.5-VL-7B, retaining only this component makes 32.5 percentage points more answers follow the donor than a norm-matched control, and removing it lowers donor-following by 57.7 points. This result was already in Section 5.4 of the original submission, and it makes the component a direct target for editing a model's answer.
  • Placing interventions. We newly show in Section 5.5 that the best layer for transferring an answer differs from the best layer for steering with a single direction: in Qwen2.5-VL-7B, the shared direction flips 91.6% of yes/no decisions at layer 17 but only 1.6% at layer 21. Methods that steer or edit VLMs should therefore choose their layer according to the type of intervention.
  • Detecting ungrounded answers. Because the component carries the answer that the image supports before later layers decode it, it offers a site at which to check whether a final answer follows the image or a language prior. We propose this check as a next step toward more grounded VLMs.

We added these implications as a new "Implications" paragraph in the new Discussion section, Section 6. Therefore, our findings tell model developers where to intervene and how to check whether an answer is grounded in the image.


Thank you again for your valuable feedback, which has helped strengthen the manuscript. We hope that our response has addressed your concerns. We kindly ask you to reconsider your evaluation or provide further feedback.

Best,

Authors

Thanks to the authors for the rebuttal. I will keep my positive rating.

AuthorsOpenReview ↗

Thank you!

Dear Reviewer HMm7,

Thank you for reading our response and for keeping your positive rating. Your suggestions on visual confounds, spatial questions, and model differences led to analyses that clearly strengthened the paper, and we will keep them in the final version.

Best, Authors

Reviewer_PvPwOpenReview ↗
Rating
2
Confidence
1
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper analyzes the internal influence of image features on Question Answering with VLMs. They found that in a specific middle layer, the VLM compresses image features into answers; in later layers, the answer features are recalculated for answering. To test this, with the same question, the authors replace the features of a specific layer with those of a donor image and observe a shift in the answer if the answer is opposite, or a shift in the confidence score if the answer is the same. They use a subset of the VQAv2 dataset and observe that all tested VLMs have this phenomenon.

Critical strengths
  1. They find that VLMs compress image features into answers from specific layers, and in the latter layers, the answer is fixed.

  2. All tested VLMs show this phenomenon across a subset of the VQAv2 dataset.

Critical weaknesses
  1. The scalability of the testing case is questionable to me. The only tested questions are yes/no from 538 VQAv2 pairs. The effects vary across different VLMs.
  2. The paper focuses mainly on findings; it is unclear to me how these findings can influence VLM performance or reveal VLM flaws.
  3. The theory and paper are hard to follow; the authors analyze the VLM using complex theory.
Questions for authors

My concerns are mainly mentioned in the weakeness.

  1. Scale of testing data.
  2. How can the findings influence VLM design.
  3. Easy to read presentation.
2 comments
AuthorsOpenReview ↗

Response to Reviewer PvPw (1/1)

Dear Reviewer PvPw,

Thank you for taking the time to review our manuscript and for recognizing as a strength our main finding that answer information is concentrated in specific layers, a pattern consistently observed across all tested VLMs. In the revised PDF, changes addressing your review are marked in green and changes addressing concerns shared with other reviewers in purple; each is tagged with the point of your review it addresses.


W1: The scalability of the testing case is questionable to me. The only tested questions are yes/no from 538 VQAv2 pairs. The effects vary across different VLMs. Q1: Scale of the test data.

A1: Thank you for raising this point. We would like to clarify that the original panel contains 538 quartets with 2,152 images, covering 371 yes/no, 74 number, and 93 open-ended questions. We added this composition to the abstract. We also added three analyses showing that the finding holds beyond this panel:

  • A second dataset. On 1,000 new GQA quartets, half of which are not yes/no, the contrast is reliably positive in all five tested settings, and in four of the five on the 342 spatial-relation quartets alone. We added this as a new paragraph in Section 4, "The contrast extends to Other questions and to GQA", with details in the new Appendix H.3.

  • Unseen answers. On the 237 pairs whose answers were never used for fitting, the component meets all six original criteria without refitting. We added this at the end of the first paragraph of Section 5.2, with details in the new Appendix H.4.

  • Visual similarity. After we adjust for image similarity and object count, the contrast stays close to its original value in all seven checkpoints. We added this as a new paragraph in Section 4, "Measured visual differences do not explain the contrast", with details in the new Appendix H.2.

Much of the variation across models reflects the layers at which they were evaluated. When the component is refitted within each model's window, Idefics3's share of the donor effect increases from 51% to 92%, while Gemma3's effects increase by a factor of 14–20. We report this in the new Section 5.5. Taken together, the finding holds beyond the original panel, and models differ mainly in where, not whether, they carry the answer summary.


W2: The paper focuses mainly on findings; it is unclear to me how these findings can influence VLM performance or reveal VLM flaws. Q2: How can the findings influence VLM design?

A2: Thank you for the comment. Our findings show where and how a VLM's answer can be changed, which has three practical implications:

  • Controlling answers. Retaining the rank-16 component makes 32.5 percentage points more answers follow the donor than a matched control, as reported in Section 5.4 of the original submission.

  • Choosing intervention layers. We newly show in Section 5.5 that answer transfer and steering work best at different layers, so the intervention layer should match the intervention type.

  • Improving interpretability studies. Without a same-answer control, image-content effects in early layers can be mistaken for answer effects, as Figure 3b shows. We recommend this control for future VLM patching studies.

We added these implications as a new "Implications" paragraph in the new Discussion section, Section 6.


Q3: Presentation.

A3: Thank you for the feedback. We revised the paper so that readers can follow it without the notation:

  1. We organized the introduction around four research questions, shown as bold headings in Section 1.
  2. We added plain-language definitions of the conditions at the end of the first paragraph of Section 3.
  3. We rewrote the opening and "Summary" paragraphs of Section 4 to state what each analysis tests and what remains open, and we moved the statistical details of Section 3 to Appendix B.

All changes for your review are highlighted in green in the revised PDF.


Thank you again for your valuable feedback, which has helped strengthen the manuscript. We hope that our response has addressed your concerns. We kindly ask you to reconsider your evaluation or provide further feedback.

Best,

Authors

AuthorsOpenReview ↗

Gentle Reminder

Dear Reviewer PvPw,

Thank you again for your valuable feedback and engagement. We believe our comprehensive replies have fully addressed all of your concerns. If any questions remain, please let us know and we will clarify immediately. Otherwise, if our responses have resolved the issues, we would be grateful for your consideration in updating the evaluation accordingly.

With sincere appreciation,

The authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary Response by Authors

Dear Reviewers, ACs, SACs, and PCs,

We sincerely thank all reviewers for their careful and constructive feedback, which has helped strengthen the scientific positioning, empirical support, and clarity of our manuscript. We summarize the key points from the discussions below to facilitate the final assessment.

Recognized Strengths:

Reviewer HMm7 finds our controlled design insightful, the experiments extensive, and the presentation clear, and has explicitly confirmed that they will maintain their positive rating after the rebuttal. Reviewer n3Vn acknowledges that the experimental design is sensible, the experiments reasonably support our conclusions, and the additional experiments improve the study's completeness. Reviewer PvPw recognizes the phenomenon observed across different VLMs. All three reviewers assess soundness as having at most minor or fixable concerns.

Main Contributions:

Our work goes beyond locating visual-information pathways to investigate what later layers read from the text-mediated visual pathway. By holding the question fixed and comparing magnitude-matched same-answer and different-answer interventions, we distinguish answer-aligned effects from those of other image content. In early layers, both interventions perturb the output similarly, whereas within the middle-layer window, different-answer interventions have substantially larger effects (Figure 3b). Furthermore, same-answer donors transmit 33–67% of their answer strength across four checkpoints, revealing a graded answer-level signal rather than a binary one. These findings establish a mechanistic distinction that complements and extends prior studies on information flow and layer-wise processing.

Main Revisions and Additional Evidence:

All revisions have been incorporated into the uploaded PDF, with reviewer-specific color highlighting and point-level annotations. Page 16 provides the Summary of Revisions, and Appendix H is the full additional analyses. The major changes are as follows:

  1. Generality across datasets and tasks (§4, Appendix H.3): We clarified that the original 538 VQAv2 quartets contain 2,152 images spanning yes/no, number, and Other questions. We added 1,000 GQA quartets, including 501 non-yes/no quartets. The answer contrast is consistently positive across all five tested model–layer settings and in four of five settings on 342 spatial-relation quartets.

  2. Visual confound controls (§4, Appendix H.2): Adjusting for image similarity, object count, and available object-size annotations leaves the main contrast largely unchanged across seven checkpoints. Controlled synthetic scenes independently reproduce the expected pattern in four of five model–layer settings.

  3. Layer dependence and architectural differences (§§4–5.5, Appendices H.1 and H.5): Refitting the component inside the middle-layer window increases the retained share of the donor effect from 51% to 92% in Idefics3, while Gemma3's projected-source effect increases approximately fourteenfold. Additional token-position interventions reveal that Qwen2.5-VL-7B concentrates the answer contrast in question tokens, whereas Idefics3 also carries it in image tokens.

  4. Causal functionality of the learned component (§5, Appendices H.4 and H.6): The rank-16 component selectively retains and removes donor influence against magnitude-matched controls, including on 237 evaluation pairs with answer strings absent from fitting. In Qwen2.5-VL-7B, retaining the component increases generated donor-following by 32.5 percentage points over the matched control. Output-letter crossing further shows that the intervention transfers the donor's semantic answer rather than merely its output letter.

  5. Contribution positioning, clarity, and implications (§§1–2, 6): We clarified the distinction from prior information-flow studies, reorganized the introduction around four interconnected research questions, simplified the experimental explanations, and added a Discussion section that covers intervention-layer selection, answer steering, and methodological implications for VLM interpretability.

We appreciate Reviewer HMm7's continued positive assessment and Reviewer n3Vn's acknowledgment of the additional experiments. We also recognize that Reviewer n3Vn maintains reservations concerning novelty and scientific significance. While we respect differences in scientific judgment, we firmly believe that our work makes multiple novel, scientifically meaningful contributions. In particular, our contribution is not merely a variation of activation patching or another observation of middle-layer processing. Rather, we establish a controlled causal distinction between answer-aligned and other image-content effects, together with evidence that a compact, behaviorally active component carries the answer-aligned signal. We believe these findings provide meaningful mechanistic insights beyond those established in prior studies.

Best,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.