- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies how Qwen3-VL decides which image region to describe next, using a synthetic strip-of-panels retrieval task (shape + letter per panel). Using donor/receiver patching, it describes a pipeline as follow:
- Binding heads at the shape word compute pointers to the matching panels.
- Relay heads carry the pointer to the answer position.
- In the list setting, the last written letter is re-bound to its panel, and that panel becomes a "cursor" ("done up to panel x").
- A distributed select step picks the next candidate after the cursor, or stops.
- Gaze heads read the region and return a concept that is mostly independent of language. Language heads and the late MLPs turn it into a word in the question's language.
There are generalization checks across model sizes (2B–32B), panel-free canvases, other prompts and layouts, COCO, and four languages.
Critical strengths
The experimental design is the best part in this paper. Building pairs so that "no effect", "copied answer", "moved place", "moved request" and "moved layout" each predict a different letter makes the patching results much easier to trust than the usual logit-diff plots. Held-out splits, a fixed rule for choosing K, and same-layer random-head controls throughout also help a lot. The progress/cursor result is the genuinely new piece for me. The binding and pointer part overlaps quite a bit with Assouel et al., Cui et al., and Gandikota & Bau. The count -> cursor experiment in 5.2 is neat, and it's a nice idea that the cursor is rebuilt from the last written label rather than carried forward. I hadn't seen this shown for VLMs before.
Critical weaknesses
- The framing outruns the evidence, I think. The title and intro talk about how VLMs plan what to describe, i.e. captioning. But the cursor is defined over a 1D reading order (left-to-right, or row order on the canvas, where the authors even filter layouts so the order is unambiguous). Real captions aren't ordered spatially; they follow salience or semantics. The COCO section only checks a few things (repeat when the comma can't see A, attention ratios), and the authors say select can't be localized there. So I'm not sure "done up to panel x" is how captioning works, or whether it's specific to list-reading prompts. I'd like this either tested more directly or scoped down in the claims.
- "Holds across model sizes" seems overstated (thoughts?). The cursor rebind moves 85.5% on 8B but only 26% on 32B (50 pairs), 23.5% on 4B, and 10.5% on 2B. The smaller models name the visited panel about as often as the position after it. On 2B, concept and language aren't separated at all (55.7% copied words). That reads to me as "a clean version at 8B and a partial one elsewhere", which is fine, but it isn't what the abstract says. Relatedly, select has no small set of heads; it's localized to a layer band. So one of the four main parts is characterized much more loosely than the others.
A few smaller things I noted here. Good if authors can check them out:
- Several components are sufficient but not necessary. For example, blinding the gaze heads still leaves 86.5% correct. "Language-free concept" might be better phrased as an English/Chinese-dominant latent, since that's what the logit lens shows.
- On the object task, the gaze heads move the concept only 9.8% of the time.
Questions for authors
- If the prompt asks for a different order (let's say, right to left, or "most salient first"), does the cursor still behave like an ordinal position in that order? This would really help separate a planning mechanism from a reading-order habit, and it would change my view of weakness 1.
- At 32B the rebind gives only 26%. Is that a sample-size issue (50 pairs), or is the cursor carried somewhere else in larger models? Did you try searching heads at the label token on 32B specifically?
- Can the mechanism explain real failure cases, like repeated objects or premature stopping in captions? For example, do the repetition errors in COCO line up with weak label-to-region binding? This isn't required, but it would make the paper much more useful beyond interpretability.
- Clarification: the "binding heads" are sometimes 16 and sometimes 24 (the union). Could you keep this consistent in the main text? It confused me a few times.
5 comments
Response to Reviewer K2xS (2/2)
W3. Smaller points.
-
Why do answers survive after blocking the gaze heads? Surprisingly, the redundancy does not come from another set of heads at the colon token. The tokens between the shape token and the colon also read the letter from the image. Blocking both paths leaves only 2.6% of answers (Sec. 6.1).
-
"Language-free concept". We agree and changed the wording. Section 6.2 now says that the gaze heads return the concept, not the word, and that their top logits are always English or Chinese, whatever the language of the question.
-
The gaze heads move the object in only 9.8%. We find that, based on the type of visual content, the model uses a specialized set of gaze heads to read it. The 9.8% is on the object task for the 32 gaze heads that we found on the letter task, where the content is text. When we repeat the head search on the object task itself, its top 32 heads move the object in 86.1% of cases, against 0.0% for random heads, and 14 of the 32 gaze heads are among them. The language heads are not specific in this way: the 16 language heads of the paper still move the language in 87.6% of cases on this task, and the object in 0.0%. We added this discussion to Section 6 of the main paper, with the details in Appendix F. This agrees with the Gaze Heads work (Gandikota & Bau, 2026, "Gaze Heads: How VLMs Look at What They Describe"), which shows that individual gaze heads specialize in fetching different types of visual content.
Q3. Can the mechanism explain real failures? Yes, for the error the model makes most. The most common error made by the 8B model is skipping an answer. When the model skips a correct answer, binding and the cursor are intact (100.0% and 92.1%). The error most commonly occurs at the select step: the gaze heads attend most to the correct next panel in only 25.4% of skips, against 95.6% of correct answers (App. E).
Q4. 16 or 24 binding heads. This inconsistency is now fixed: the updated paper uses the 16 binding heads throughout. Where an experiment adds the query and key heads, the paper says so and names the union of 18 heads (App. B).
We hope this answers your concerns. We are happy to answer any further questions, or to run additional experiments if time permits. We thank you again for your suggestions, which added new insights to our findings and made the paper more thorough and rigorous.
Really good job in the rebuttal
Thanks for the detailed response and all the new experiments. Honestly more than I expected in a rebuttal period. If these results are reproducible (I do really hope so), it would be a really nice contribution and should be emphasized in the camera-ready version.
- First of all, the right-to-left result is the one that changes my mind. This is a real planning step and not just a reading habit. I also appreciate that you scaled back the model-size claims instead of defending them (which is my initial thought)
- A few things I'm still not fully sure about. It's fine to keep these as open questions, but probably worth something we can consider more?
-- Under the explicit right-to-left prompt, the heads build progress in the direction in 66% of cases, lower than 73.8% under the original prompt. Any idea why the mechanism gets weaker exactly when the order is requested? Is the rest of the effect carried somewhere else?
-- On COCO, it's nice that size matters, but the claim that salience enters at the same step as the order instruction is still a guess. I'd phrase it as a hypothesis in the paper rather than a finding itself.
-- For the 32B, 68% is below the 80% threshold you use for head selection everywhere else. How many held-out cases is that on? Probably some confidence interval-ish thing around here will justify.
-- The skip-error analysis is a nice addition, but for now it's on the synthetic task and based on attention rather than a causal intervention. My question was more about COCO-style repetition or early stopping. I meant it's fine to leave this out, but please don't overstate it as explaining "real failures".
-- A small one: you now mention a "union of 18 heads", but your original tables had 8 query + 16 key heads = 17 (am I missing something?), and 24 once the output heads were added. Could you double-check which sets this refers to?
-- Besides, I also noticed a couple of number mismatches in the original draft that I didn't raise before: the Sec. 5 progress percentages vs. Table 18 for the 8B, and Table 6 reporting 100% "next unsaid place" while Fig. 16 shows the model sometimes going back. Worth a quick check in the revision.
Overall, yes, I think the paper is in a much better place, and I'll raise my score to weak accept. Not entirely sure about an award for this paper (to me, of course), but still, it showed good effort of doing research (much much better than other papers I read in my review batch, as it stands. And this rebuttal made my day.).
Response to Reviewer K2xS (1/2)
We thank Reviewer K2xS for the detailed and constructive review. We are glad you found our experimental design easy to trust and our progress result to be novel. We ran the experiments you proposed and found several interesting results: a new step in the plan that reads the requested order, the heads that set the cursor in the 32B, and the step that fails when the model skips an answer. We genuinely believe that your suggestions made the paper stronger, and that the added experiments add a lot of value to it. We answer each point below in detail.
W1, Q1. Is the cursor a planning mechanism or a reading-order habit? The cursor is not a reading-order habit, and we found a new step in the plan when we prompt the model to list in a specific order. When we add the sentence "List them from right to left." to the prompt, the model lists from right to left in most cases. We found that the heads we discovered earlier build progress in the requested direction in 66.0% of cases under explicit prompting (73.8% under the original prompt and 0.0% for random heads). The direction comes from the instruction: 16 heads in layers 18-23 read it before the cursor forms. Blocking these heads from looking back at the instruction returns the list to the default left-to-right order in 98.4% of cases. Conversely, under explicit prompting, blocking every head except these 16 from reading the instruction tokens keeps the right-to-left order in 98.4% of cases. We added this new discovery to Section 5.4 and Appendix G.
The cursor also does not need our layout filter. On the scattered layouts that the filter rejected, with the order taken from the model's own list, the binding heads still set the cursor in 82.0% of cases, against at most 2.0% for random heads (App. G).
On COCO, the order is not left to right: the larger object comes first in 70.8% of pairs, and an instruction such as "start with the most prominent one" or "from left to right" changes the order in the requested direction. The steps we can test on photographs carry over. When the comma token cannot attend to the object just named, the model names it again in 79.6% of lists, and the gaze heads at the comma token attend more to the object named next in 83.6%. Our causal tests of the cursor are on list questions, and the introduction now states which findings carry over to photographs.
W2, Q2. Does the mechanism hold across model sizes? Where does the 32B set the cursor? The 32B sets the cursor at the same token as the 8B, but with more heads. We reran the search over all 4,096 heads at the answer token of the 32B. It selects 64 heads, 1.6% of the model, which move the cursor in 68.0% of held-out cases, against 0.0% for random heads. The 16 binding heads alone give 22.0% on the same cases, which is the low rate you noticed. Nine of them are among the 64, and the others sit in the layers around them. The cursor itself is clean in the 32B: the copied comma state gives it in 100% of pairs at layer 48 of 64 (App. G).
For the 2B and 4B, we agree that our claim that the mechanism holds across model sizes was too strong. The 2B is much weaker at the task itself: it lists the second letter correctly in only 48.0% of pairs. The binding, relay, and gaze heads appear at every size, but the cursor is clean only on the 8B and 32B. The 2B and 4B hold the count and the visited panel about as often as the cursor, and on the 2B, concept and language are not yet separate. We updated our writing accordingly.
We do not find that select is localized to a set of attention heads (our updated Section 5.4 states this). It is still localized causally, to the comma token's residual in layers 20-25. After we move the cursor, holding this residual at the target's own values at layer 23 or later brings the target's answer back in 99.4% of cases. Holding only the gaze heads' queries brings it back in 99.8%, so the gaze heads read the result of select.