acceptodds
Under review as a conference paper at ICLR 2027

How VLMs Plan What's Next

Abstract

When a vision-language model describes an image, it names one object at a time and it rarely repeats itself, yet how the model plans this order is not obvious. We characterize the mechanism a VLM uses to decide where to look before it looks. The mechanism has four main parts. First, binding heads match the words of the question against the image and turn them into pointers to every matching image region. Second, the model binds each written answer to the region it came from, and turns it into a progress signal: a cursor, the position in the image up to which the list has been written. Third, a select operation chooses the nearest unwritten pointer after the cursor, or stops generating. Finally, gaze heads read the image at the chosen region and return a language-free concept, which language heads and the final MLPs turn into a word in the language of the question. We validate each part through causal patching in Qwen3-VL models on a synthetic visual retrieval task, pairing donor and receiver runs so that every alternative explanation predicts a different answer. We further show that the mechanism holds across model sizes, input formats and languages.

Primary area:
interpretability and explainable AI
Keywords:
Interpretability, Vision Language Models, Muti Modal

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_K2xSOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how Qwen3-VL decides which image region to describe next, using a synthetic strip-of-panels retrieval task (shape + letter per panel). Using donor/receiver patching, it describes a pipeline as follow:

  1. Binding heads at the shape word compute pointers to the matching panels.
  2. Relay heads carry the pointer to the answer position.
  3. In the list setting, the last written letter is re-bound to its panel, and that panel becomes a "cursor" ("done up to panel x").
  4. A distributed select step picks the next candidate after the cursor, or stops.
  5. Gaze heads read the region and return a concept that is mostly independent of language. Language heads and the late MLPs turn it into a word in the question's language.

There are generalization checks across model sizes (2B–32B), panel-free canvases, other prompts and layouts, COCO, and four languages.

Critical strengths

The experimental design is the best part in this paper. Building pairs so that "no effect", "copied answer", "moved place", "moved request" and "moved layout" each predict a different letter makes the patching results much easier to trust than the usual logit-diff plots. Held-out splits, a fixed rule for choosing K, and same-layer random-head controls throughout also help a lot. The progress/cursor result is the genuinely new piece for me. The binding and pointer part overlaps quite a bit with Assouel et al., Cui et al., and Gandikota & Bau. The count -> cursor experiment in 5.2 is neat, and it's a nice idea that the cursor is rebuilt from the last written label rather than carried forward. I hadn't seen this shown for VLMs before.

Critical weaknesses
  • The framing outruns the evidence, I think. The title and intro talk about how VLMs plan what to describe, i.e. captioning. But the cursor is defined over a 1D reading order (left-to-right, or row order on the canvas, where the authors even filter layouts so the order is unambiguous). Real captions aren't ordered spatially; they follow salience or semantics. The COCO section only checks a few things (repeat when the comma can't see A, attention ratios), and the authors say select can't be localized there. So I'm not sure "done up to panel x" is how captioning works, or whether it's specific to list-reading prompts. I'd like this either tested more directly or scoped down in the claims.
  • "Holds across model sizes" seems overstated (thoughts?). The cursor rebind moves 85.5% on 8B but only 26% on 32B (50 pairs), 23.5% on 4B, and 10.5% on 2B. The smaller models name the visited panel about as often as the position after it. On 2B, concept and language aren't separated at all (55.7% copied words). That reads to me as "a clean version at 8B and a partial one elsewhere", which is fine, but it isn't what the abstract says. Relatedly, select has no small set of heads; it's localized to a layer band. So one of the four main parts is characterized much more loosely than the others.

A few smaller things I noted here. Good if authors can check them out:

  • Several components are sufficient but not necessary. For example, blinding the gaze heads still leaves 86.5% correct. "Language-free concept" might be better phrased as an English/Chinese-dominant latent, since that's what the logit lens shows.
  • On the object task, the gaze heads move the concept only 9.8% of the time.
Questions for authors
  1. If the prompt asks for a different order (let's say, right to left, or "most salient first"), does the cursor still behave like an ordinal position in that order? This would really help separate a planning mechanism from a reading-order habit, and it would change my view of weakness 1.
  2. At 32B the rebind gives only 26%. Is that a sample-size issue (50 pairs), or is the cursor carried somewhere else in larger models? Did you try searching heads at the label token on 32B specifically?
  3. Can the mechanism explain real failure cases, like repeated objects or premature stopping in captions? For example, do the repetition errors in COCO line up with weak label-to-region binding? This isn't required, but it would make the paper much more useful beyond interpretability.
  4. Clarification: the "binding heads" are sometimes 16 and sometimes 24 (the union). Could you keep this consistent in the main text? It confused me a few times.
5 comments
AuthorsOpenReview ↗

Response to Reviewer K2xS (1/2)

We thank Reviewer K2xS for the detailed and constructive review. We are glad you found our experimental design easy to trust and our progress result to be novel. We ran the experiments you proposed and found several interesting results: a new step in the plan that reads the requested order, the heads that set the cursor in the 32B, and the step that fails when the model skips an answer. We genuinely believe that your suggestions made the paper stronger, and that the added experiments add a lot of value to it. We answer each point below in detail.

W1, Q1. Is the cursor a planning mechanism or a reading-order habit? The cursor is not a reading-order habit, and we found a new step in the plan when we prompt the model to list in a specific order. When we add the sentence "List them from right to left." to the prompt, the model lists from right to left in most cases. We found that the heads we discovered earlier build progress in the requested direction in 66.0% of cases under explicit prompting (73.8% under the original prompt and 0.0% for random heads). The direction comes from the instruction: 16 heads in layers 18-23 read it before the cursor forms. Blocking these heads from looking back at the instruction returns the list to the default left-to-right order in 98.4% of cases. Conversely, under explicit prompting, blocking every head except these 16 from reading the instruction tokens keeps the right-to-left order in 98.4% of cases. We added this new discovery to Section 5.4 and Appendix G.

The cursor also does not need our layout filter. On the scattered layouts that the filter rejected, with the order taken from the model's own list, the binding heads still set the cursor in 82.0% of cases, against at most 2.0% for random heads (App. G).

On COCO, the order is not left to right: the larger object comes first in 70.8% of pairs, and an instruction such as "start with the most prominent one" or "from left to right" changes the order in the requested direction. The steps we can test on photographs carry over. When the comma token cannot attend to the object just named, the model names it again in 79.6% of lists, and the gaze heads at the comma token attend more to the object named next in 83.6%. Our causal tests of the cursor are on list questions, and the introduction now states which findings carry over to photographs.

W2, Q2. Does the mechanism hold across model sizes? Where does the 32B set the cursor? The 32B sets the cursor at the same token as the 8B, but with more heads. We reran the search over all 4,096 heads at the answer token of the 32B. It selects 64 heads, 1.6% of the model, which move the cursor in 68.0% of held-out cases, against 0.0% for random heads. The 16 binding heads alone give 22.0% on the same cases, which is the low rate you noticed. Nine of them are among the 64, and the others sit in the layers around them. The cursor itself is clean in the 32B: the copied comma state gives it in 100% of pairs at layer 48 of 64 (App. G).

For the 2B and 4B, we agree that our claim that the mechanism holds across model sizes was too strong. The 2B is much weaker at the task itself: it lists the second letter correctly in only 48.0% of pairs. The binding, relay, and gaze heads appear at every size, but the cursor is clean only on the 8B and 32B. The 2B and 4B hold the count and the visited panel about as often as the cursor, and on the 2B, concept and language are not yet separate. We updated our writing accordingly.

We do not find that select is localized to a set of attention heads (our updated Section 5.4 states this). It is still localized causally, to the comma token's residual in layers 20-25. After we move the cursor, holding this residual at the target's own values at layer 23 or later brings the target's answer back in 99.4% of cases. Holding only the gaze heads' queries brings it back in 99.8%, so the gaze heads read the result of select.

AuthorsOpenReview ↗

Response to Reviewer K2xS (2/2)

W3. Smaller points.

  • Why do answers survive after blocking the gaze heads? Surprisingly, the redundancy does not come from another set of heads at the colon token. The tokens between the shape token and the colon also read the letter from the image. Blocking both paths leaves only 2.6% of answers (Sec. 6.1).

  • "Language-free concept". We agree and changed the wording. Section 6.2 now says that the gaze heads return the concept, not the word, and that their top logits are always English or Chinese, whatever the language of the question.

  • The gaze heads move the object in only 9.8%. We find that, based on the type of visual content, the model uses a specialized set of gaze heads to read it. The 9.8% is on the object task for the 32 gaze heads that we found on the letter task, where the content is text. When we repeat the head search on the object task itself, its top 32 heads move the object in 86.1% of cases, against 0.0% for random heads, and 14 of the 32 gaze heads are among them. The language heads are not specific in this way: the 16 language heads of the paper still move the language in 87.6% of cases on this task, and the object in 0.0%. We added this discussion to Section 6 of the main paper, with the details in Appendix F. This agrees with the Gaze Heads work (Gandikota & Bau, 2026, "Gaze Heads: How VLMs Look at What They Describe"), which shows that individual gaze heads specialize in fetching different types of visual content.

Q3. Can the mechanism explain real failures? Yes, for the error the model makes most. The most common error made by the 8B model is skipping an answer. When the model skips a correct answer, binding and the cursor are intact (100.0% and 92.1%). The error most commonly occurs at the select step: the gaze heads attend most to the correct next panel in only 25.4% of skips, against 95.6% of correct answers (App. E).

Q4. 16 or 24 binding heads. This inconsistency is now fixed: the updated paper uses the 16 binding heads throughout. Where an experiment adds the query and key heads, the paper says so and names the union of 18 heads (App. B).

We hope this answers your concerns. We are happy to answer any further questions, or to run additional experiments if time permits. We thank you again for your suggestions, which added new insights to our findings and made the paper more thorough and rigorous.

Reviewer_K2xSOpenReview ↗

Really good job in the rebuttal

Thanks for the detailed response and all the new experiments. Honestly more than I expected in a rebuttal period. If these results are reproducible (I do really hope so), it would be a really nice contribution and should be emphasized in the camera-ready version.

  1. First of all, the right-to-left result is the one that changes my mind. This is a real planning step and not just a reading habit. I also appreciate that you scaled back the model-size claims instead of defending them (which is my initial thought)
  2. A few things I'm still not fully sure about. It's fine to keep these as open questions, but probably worth something we can consider more?

-- Under the explicit right-to-left prompt, the heads build progress in the direction in 66% of cases, lower than 73.8% under the original prompt. Any idea why the mechanism gets weaker exactly when the order is requested? Is the rest of the effect carried somewhere else?

-- On COCO, it's nice that size matters, but the claim that salience enters at the same step as the order instruction is still a guess. I'd phrase it as a hypothesis in the paper rather than a finding itself.

-- For the 32B, 68% is below the 80% threshold you use for head selection everywhere else. How many held-out cases is that on? Probably some confidence interval-ish thing around here will justify.

-- The skip-error analysis is a nice addition, but for now it's on the synthetic task and based on attention rather than a causal intervention. My question was more about COCO-style repetition or early stopping. I meant it's fine to leave this out, but please don't overstate it as explaining "real failures".

-- A small one: you now mention a "union of 18 heads", but your original tables had 8 query + 16 key heads = 17 (am I missing something?), and 24 once the output heads were added. Could you double-check which sets this refers to?

-- Besides, I also noticed a couple of number mismatches in the original draft that I didn't raise before: the Sec. 5 progress percentages vs. Table 18 for the 8B, and Table 6 reporting 100% "next unsaid place" while Fig. 16 shows the model sometimes going back. Worth a quick check in the revision.

Overall, yes, I think the paper is in a much better place, and I'll raise my score to weak accept. Not entirely sure about an award for this paper (to me, of course), but still, it showed good effort of doing research (much much better than other papers I read in my review batch, as it stands. And this rebuttal made my day.).

Reviewer_Xa9rOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how VLMs select successive image regions during visual retrieval. Activation-patching experiments identify mechanisms for binding requests to locations, tracking progress through a spatial cursor, and retrieving subsequent answers. It also investigates the separation between answer content and output language. Experiments primarily use Qwen3-VL-8B, with extensions across model sizes, prompts, layouts, and natural images.

Critical strengths

Carefully designed interventions. The donor–receiver pairs distinguish transfer of the request, layout, selected location, and answer content. This makes the activation-patching results informative about competing explanations.

Informative analysis of progress tracking. The experiments distinguishing a count, visited locations, and an ordinal cursor, together with interventions rebinding generated labels to image locations, provide a concrete account of how prior answers guide subsequent retrieval in the studied task.

Critical weaknesses

The experiments address a narrower question than the paper’s framing. The paper opens by asking how a VLM decides which part of an image to describe next. However, the main experiments specify a target shape and investigate how the model retrieves matching regions in sequence. Appendix E.5 extends the analysis to object enumeration, which remains a different task from open-ended image description. The experiments therefore leave unclear how the model chooses what to mention first and how to order subsequent content under “describe the image.” The framing should more clearly reflect this scope.

The intervention definitions and experimental coverage are insufficiently clear in the main text. Lines 158–159 list residual, MLP, head-output, and query/key/value patches without adequately explaining what is replaced, at which token positions, and what remains unchanged. These details are central to interpreting the results, not merely implementation details. Appendix A.4 helps, but a concise schematic or worked example is needed in the main text. A consolidated experimental overview or even a figure would make the scope of the evidence much easier to assess.

Minor Comment - line 211 refers to the marker token in Figure 2b as “star,” while the figure labels it “triangle.” Please make the examples consistent, it hurts readability.

Questions for authors
  1. In the main experiments, binding heads use the named shape to identify relevant regions. Under “describe the image,” there is no equivalent target-shape token. Do you have evidence for how the first region to describe are selected in this setting, or is this outside the scope of the proposed mechanism?

  2. How much this analysis is sensitive to prompt? Under prompt B in E. 4, binding and relay patches are less effective, while gaze-head interventions remain strong (Tables 13–14). Please clarify what this implies for the generality of the proposed mechanism.

2 comments
AuthorsOpenReview ↗

Response to Reviewer Xa9r

We thank Reviewer Xa9r for the thoughtful and constructive review. We are glad you found our interventions carefully designed. Following your suggestions, we revised the framing, made the interventions clear in the main text, and ran a new experiment on the wording of the prompt that explains the weaker case. We answer each point below in detail.

W1. The experiments address a narrower question than the framing. We have added more clarity and revised the framing. The introduction now says that an open-ended description is hard to study causally, since it is hard to control which objects the model describes or in what order. So we design a synthetic retrieval task that lets us control each part of the mechanism on its own. The introduction then states what we test beyond it: other layouts, other prompt wordings, and list questions on natural photographs.

W2. Intervention definitions and experimental coverage. We updated the Intervention paragraph of Section 3 with clear information about the patching and the exact intervention details. Next to the worked example of a source and target pair, it now states that every patch replaces one activation of the target run with that of the source run, at one token and one layer, and leaves everything else in the target run unchanged. It names the three levels we patch (the residual vector, the output of an MLP, and the output of an attention head) and says what each head patch changes: a query patch changes what the head looks for, a key patch lets it read the source's image keys, and a value patch changes what it brings back. Following your suggestion, we added a new section in Appendix A that lists every experiment of the main paper with the token where the patch is applied, the activation it replaces, the answer the tested hypothesis predicts, and the result. These changes helped improve the prose and clarity of our paper.

W3. "star" in the text and "triangle" in Figure 2b. Thank you, this is fixed throughout the paper.

Q1. How is the first region selected under "describe the image"? Without a named target, the binding heads cannot base their pointer resolution on text anymore. How an open-ended description picks its first region is outside the scope of our experimental setup. However, the closest evidence we have is on COCO list questions that name a category of objects to list, but without any defined reading order. There, the first item is the largest annotated object in 63.2% of lists, against 40.6% by chance, and the larger object comes first in 70.8% of pairs. An instruction such as "start with the most prominent one" changes the order in the requested direction. So on photographs the order follows size and the prompt, not position (App. G).

Q2. How sensitive is the analysis to the prompt? The mechanism does not depend on the wording. The weaker case has a definite cause: that wording goes on for two sentences after the shape token, and our binding patch acted on the shape token alone. With the same heads discovered in the paper, the binding heads patched at the shape token alone move the pointer in 52.0% of cases. Patched at the shape token and the tokens after it, they move it in 87.0%, against 0.0% for random heads. The relay heads then carry the pointer to the colon in 88.0% of cases, against 58.0% before. So with a longer question, the model spreads out its binding resolution over more tokens. The gaze-head edits stayed strong because the gaze heads act at the colon token under any wording. This experiment adds more clarity to what we earlier believed was a failure case, and we added it to Appendix G.

We hope this answers your concerns. We are happy to answer any further questions, or to run additional experiments if time permits. We thank you again for your suggestions, which added new insights to our findings and made the paper more thorough and rigorous.

Thank you for the detailed response and additional experiments. The new results address my concerns about prompt sensitivity, and I appreciate the clarified framing and intervention details.

Overall, I believe the authors have satisfactorily addressed my questions and concerns. I maintain my positive assessment and original score.

Reviewer_Q3Q3OpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper uses activation patching to find mechanisms within VLMs of how they plan what to describe next in an image. They conduct extremely thorough experiments to plot out exactly, step by step, how VLMs do so: binding heads match the request against the image and point to the matching panel, relay heads carry the pointer to the token before the answer is generated, gaze heads use it to retrieve the answer from the matching panel, the model records which panel it has reached in the sequence, and then selects other matches if present.

The authors conduct interesting and extremely thorough experiments: not only controls for each experiment, but also experiments in different languages, with and without color, with and without panel lines, on real images, etc.

Critical strengths

Overall, this is an extremely strong paper, and I strongly recommend it not only for acceptance, but also for consideration of an award. It sets out to address a challenging question, and does so in a thorough and sound manner. I believe this paper will provide significant insight to the community, and inspire much future work.

  1. The research question is interesting, and goes beyond existing research in a meaningful and challenging way.
  2. The experiments are very sound, tightly controlled, and very thorough, and the additional experiments are very extensive. The appendix contains many variations of the experiments, e.g., with color, with different prompts, without the panel lines, with natural images, etc.
  3. The findings are very interesting: I'm surprised the authors were able to narrow these mechanisms down so neatly, and this paper provides deep insight into VLMs' planning mechanisms. I especially liked the language-free concept findings, and how it translated into different languages (Sec 6.2, Appendix).
  4. The writing is very clear.
  5. I would actually go so far as to say the authors' use of AI agents to do such thorough experimentation, including suggestion of tight controls, is a very good use case of how AI agents should be used in research.
Critical weaknesses

I could not find any "critical weaknesses", so to speak. However, here are some minor suggestions I have for the authors.

  1. While the writing throughout the paper is very clear, I would consider clarifying the first use of the word "patch" alone to mean activation patching rather than, perhaps, an image patch, just to make the paper more accessible to readers without as much of a background in mechanistic interpretability. Also, "donor" and "receiver" as "activation patch donor/receiver", and defining "comic images".
  2. While I understand the appeal of a broad title, I would suggest adding slightly more information to the title to more accurately reflect exactly the questions being addressed in the paper. "How VLMs Plan What's Next" could as easily be applied to long-horizon task planning, rather than the focused questions in this paper.
  3. I understand the challenges of limited space, but I would recommend plugging some of the experiments in the appendix, such as the natural images experiments, more clearly in the main paper.
Questions for authors

This is a question raised by the work. It would not materially affect my assessment, but I am curious about the authors' opinion about this based on their findings and experiments.

  1. The VLM seems to scan from left to right (and up to down without panel lines). Where and how does salience come into play? I would have thought that VLMs would attend to the most salient part of the image first, then less-salient parts, because that is typically the order in which objects are mentioned in the image-text pairs they are trained on. While this may not apply to the synthetic images, what about the COCO experiments?
1 comment
AuthorsOpenReview ↗

Response to Reviewer Q3Q3

We thank Reviewer Q3Q3 for the generous and encouraging review. We are glad you found our research question meaningful and our experiments sound and thorough, and we are honored by your recommendation. Your question on salience led us to several new experiments, which we added to the paper. We answer each point below in detail.

W1. Defining "patch", the two runs, and "comic images". Thank you. We updated the paper to define these terms, and we rewrote the sections so that the exact experiment setup is stated before discussing the results. We changed the terms donor and receiver to source and target, to match the standard terminology of prior work. Section 3 now defines activation patching where we first use it. Section 6 now explains that the published Gaze Heads were found on a model describing comic book panels.

W2. A more informative title. We agree with your perspective: "plan" alone can also suggest long-horizon task planning. We believe both readings are fair. We kept our original title because what the model plans to describe next is exactly the question we answer in the paper, even though we answer it in a controlled setting. To address your concern, the introduction now states the scope much earlier. We also added experiments that show how the mechanism carries over beyond our setup, to other layouts, other prompt wordings, other model families, and natural photographs.

W3. Natural-image experiments in the main text. The main COCO results are now in the main text. Section 5.4 reports that on COCO list questions, when the comma token cannot attend to the object already described, the model names that object again in 79.6% of cases. Section 6 reports that the gaze heads at the comma token attend more to the object named next in 83.6% of lists, and that setting the language heads to their English averages turns 96.5% of non-English answers into English. The introduction also points to these experiments.

Q1. Where does salience come into play, and what about COCO? On COCO, the model does not scan from left to right. It lists larger objects first. We ran three new experiments (App. G).

  • What sets the order on COCO? The larger object comes first in 70.8% of pairs, and the first item is the largest annotated object in 63.2% of lists, against 40.6% by chance. Left to right gives 53.1%, which is close to chance, and mirroring the photograph reverses only 13.4% of pairs.

  • Is it exclusively size based saliency? No. On synthetic canvases with shapes of different sizes, the model still lists them in reading order (88.9 to 90.6% of pairs), and the larger shape comes first in ~50%, which is random. So the order on COCO likely comes from what goes with size in a real scene, such as the foreground or how easy an object is to recognize.

To find where the order enters the mechanism, we ran a new experiment on our strips based on your and Reviewer K2xS’s suggestion. We found a new hidden step in the discovered plan. With "List them from right to left." added to the prompt, 16 heads in layers 18-23 read the instruction just before the cursor forms, and set the direction of the cursor. Blocking these heads from the instruction returns the list to left to right in 98.4% of cases (Sec. 5.4). So a requested order enters the plan just before the cursor step. This suggests that salience acts at the same step, by setting where the cursor moves next.

We hope this answers your questions. We are happy to answer any further questions, or to run additional experiments if time permits. We thank you again for your suggestions, which added new insights to our findings and made the paper more thorough and rigorous.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response: Summary of Revisions and New Experiments.

We thank all the reviewers for their thoughtful and generous feedback. We are glad that the reviewers found our research "beyond existing research in a meaningful and challenging way" and our experiments "very sound, tightly controlled, and very thorough" (Q3Q3), with carefully designed interventions (Xa9r) making the results "much easier to trust", and that our progress discovery is something they "hadn’t seen this shown for VLMs before" (K2xS). We are also honored that Reviewer Q3Q3 recommends our work for an award.

Following the reviewers' suggestions, we rewrote the paper to make it easier to follow and ran seven new experiments that add new insights. We summarize the changes below:

Writing. Each section now states the question it answers and defines its experiment and metrics. We define activation patching and the comic panels where they first appear, and call the two runs of our patching experiments the source and the target (Q3Q3). The introduction now explains why we work on a synthetic task and which findings carry over to open-ended settings (Q3Q3, Xa9r, K2xS). A new appendix section lists every main-text experiment with the token it acts on, the activation it replaces, and the answer it predicts (Xa9r). Head-set names and sizes are consistent throughout (K2xS). We also folded all new experiments into the paper, each with its own discussion.

Scale and models. We reran the main-text experiments at a larger scale with confidence intervals, and all main findings hold. Beyond Qwen3-VL from 2B to 32B, we added new model families: LLaVA-OneVision-7B, InternVL3-8B, Gemma-3-12B, Pixtral-12B, and Idefics3-8B. All six models resolve the pointer at the shape token, and the cursor appears in all four that can solve the list task.

New experiments.

  • What sets the order in COCO images? (Q3Q3, Xa9r, K2xS) On COCO, the larger object comes first in 70.8% of pairs. Size alone does not set the order: on synthetic canvases with shapes of different sizes, the model still lists them in reading order. An instruction such as "from left to right" or "start with the most prominent one” changes the order in the requested direction.

  • What if the prompt asks for another order? (K2xS) We find one additional step in the plan. With "List them from right to left." added to the prompt, 16 heads in layers 18-23 read the instruction before the cursor forms, and change the direction of the cursor: instead of taking the next matching panel to the right, the model takes the next one to the left. Blocking these heads from the instruction returns the list to left to right in 98.4% of cases. Conversely, blocking every other head keeps it right to left in 98.4%.

  • Does the cursor need our layout filter? (K2xS) No. On the scattered layouts the filter rejected, the binding heads still set the cursor in 82.0% of cases, against at most 2.0% for random heads.

  • Can the discovered mechanism debug failure cases? (K2xS) When the model wrongly skips a correct answer, binding and the cursor are intact (100.0% and 92.1%). The error is at select: the gaze heads attend most to the correct next panel in only 25.4% of skips, against 95.6% of correct answers.

  • Why do answers survive even after blocking the gaze heads? (K2xS) Surprisingly, the redundancy does not come from another set of attention heads at the colon token. The tokens between the shape token and the colon also read the letter from the image. Blocking the gaze heads at the colon and these tokens from the image leaves only 2.6% of answers.

  • How does the 32B set the cursor? (K2xS) We reran the head search on the 32B, over all 4,096 heads at the answer token. It selects 64 heads, 1.6% of the model, which move the cursor in 68.0% of held-out cases, against 0.0% for random heads.

  • Does the mechanism hold under a different wording? (Xa9r) With two sentences after the shape token, the same binding heads resolve the pointer on the shape token and the tokens after it, in 87.0% of cases against 0.0% for random heads. The relay heads then carry it to the colon (88.0%).

We answer each reviewer in detail in the individual replies, and we have uploaded the revised paper. We are happy to answer any further questions, or to run additional experiments if time permits. We thank the reviewers again for their suggestions, which made the paper clearer and more rigorous.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.