acceptodds
Under review as a conference paper at ICLR 2027

Position, Not Content: What Repairs Question-First Prompting in Vision-Language Models

Abstract

Putting the question before the image rather than after it changes nothing about what a vision-language model is told, yet it changes what the model answers. Prior work by the present authors established this question-first paradox on three yes/no benchmarks and traced it to read-out rather than perception: the question does reach the image and visibly steers it, but the answer position cannot read it back, and repeating either the question or the image repairs the damage. What that work could not say is how far the effect reaches, what the repeated image actually contributes, and what it costs. We answer those three questions. Across 14 further benchmarks the paradox proves general, costing 7 points on average and as much as 18 on a single benchmark, yet it is not a law: on object detection, where the model returns boxes instead of a word, the effect reverses and asking first is the better choice. The repeated image turns out to carry almost no information. Shrinking it to 1 8 of its resolution leaves accuracy statistically unchanged while cutting its price from thousands of extra tokens to about 100, and a reversed copy works as well as a correct one, so what matters is where the tokens sit and not what they show. Breadth also marks a limit: on video, repeating the image stops helping, while repeating the question still does. A reflective prompt optimizer given the same budget improves only the ordering our diagnosis says has room left, and never catches the simpler fix, so this is not something better wording would have found. The practical consequence is immediate: put the question after the image, and where it has to come first, repeat it afterwards. The consequence for the field is that prompt layout, which papers rarely report, moves benchmarks further than the model differences those benchmarks are built to measure.

Primary area:
interpretability and explainable AI
Keywords:
Vision-Language Models, Prompt Ordering, Visual Steering, Logit Lens, Question-First Paradox, Interpretability

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_YU7YOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies a fairly simple but practically relevant prompt-formatting issue for VLMs: whether the question is placed before or after the image. The main observation is that, for several autoregressive VLMs, putting the question first can noticeably hurt accuracy even though the model receives exactly the same image and text. The proposed fix is also simple: either repeat the question after the image, or repeat the image after the question.

Critical strengths
  1. The paper provides strong controlled evidence for the prompt-layout effect. The central STI-versus-SIT comparisons preserve prompt content and token count while changing order, and the paper evaluates the effect over many datasets and output formats. This is an important question since unreported layout differences can exceed the model gaps benchmark papers aim to measure.

  2. The paper offers comprehensive analysis on this problem. The author actively probes boundaries, for example, detection reverses the ordering effect, image echoing is weak on video, POPE does not benefit, and Gemma does not show the same paradox. Reporting these limits, plus exact token costs and the GEPA comparison, makes the work more useful.

Critical weaknesses
  1. The novelty is constrained by substantial overlap with the authors’ prior workshop paper. According to the submission, the prior paper already established the paradox on three benchmarks, the read-out diagnosis, attention knockout, both echo repairs, and the reversed-image control. The new contributions are breadth, detection/video boundaries, resolution/cost analysis, automated patch visualization, and a prompt-optimizer comparison. These are useful, but the submission’s main conceptual story and intervention are not new to this paper.

  2. The claimed “position, not content” mechanism is overgeneralized. The causal analysis is performed primarily on Qwen3-VL-8B and explicitly does not explain the Gemma results, where image echoing helps without the paradox. This also raises the possibility that other VLMs, including stronger closed-source models that are not evaluated in the paper, may not exhibit the paradox at all. Furthermore, a 1/8-resolution echo still adds about 91 image tokens (and 107 total tokens) and may preserve coarse scene/object information, and spatial reversal also preserves most semantic content. Neither control is equivalent to content-free tokens.

  3. The new image/video benchmarks are reported primarily for Qwen3-VL-8B. Cross-model evidence is concentrated on NaturalBench and four selected image benchmarks, while the detailed mechanistic study is almost entirely Qwen-specific. Since Gemma does not exhibit the paradox, a broader factorial evaluation across architectures and prompt templates is necessary.

  4. The read-out probe uses cases where STI is wrong and SIT is right, a selection that necessarily produces large final-layer probability differences and may amplify attention contrasts. This subset is appropriate for studying failures, but the paper should also report the same measurements on an unconditioned sample, the reverse-disagreement set, and cases where both orderings agree to establish whether the signature is specific to the proposed failure mode.

  5. The GEPA experiment is narrow. It uses a single optimizer, a separate reflection model, small train/validation sets, and only three benchmarks. GEPA is also prohibited from changing layout, so the result mainly shows that text-only editing cannot always compensate for a positional constraint, not that “better wording” could not recover the gap.

Questions for authors
  1. Could the authors clearly distinguish the contributions of this submission from the prior workshop paper, including which claims and experiments are genuinely new?
  2. Does the paradox occur in stronger or closed-source VLMs? Have the authors tested a content-free, token-matched echo to isolate position from residual visual information?
  3. Can the authors provide broader cross-model results beyond NaturalBench and the four selected image benchmarks, especially given that Gemma does not exhibit the paradox?
  4. Do the read-out and attention patterns remain on unconditioned samples, reverse-disagreement cases, and examples where both orderings agree?
  5. Given that GEPA cannot modify prompt layout, what evidence supports the broad claim that better wording cannot recover the gap rather than a conclusion specific to GEPA?
2 comments

We thank the reviewer for precise questions and for recognizing the controlled design and the boundaries we report.

Q1. What is new relative to the workshop paper.

Claim / experimentWorkshopThis paper
Paradox on NaturalBench, POPE, Winogroundyesrecap (Tab. 1)
Perception probe, read-out probe, knockoutyesrecap (Sec. 4)
Question echo, image echo, reversed echoyesrecap (Sec. 5)
Paradox on 13 further image and video benchmarksnew (Tab. 2)
Sign reversal on detection, two modelsnew (Tab. 3)
RF20 presence: same images, opposite signnew (Tab. 5)
Image echo fails on video, question echo does notnew (Sec. 6.3)
Echo-resolution sweep (1/64 of pixels suffices)new (Tab. 4)
Exact token cost per orderingnew (Tab. 11, Fig. 2)
Automated patch-difference procedurenew (Sec. 4.2, Fig. 1)
GEPA asymmetry testnew (Sec. 7.3)
Gemma-4-31B; cross-model image benchmarksnew (Tab. 7)
Permutation and content-free echo controlsnew (rebuttal)

The workshop paper asks whether the paradox exists and why. This paper asks how far it reaches, what the repair is made of, and what it costs. We will shorten the recap so the new results take most of the main text.

Q2. Stronger models; content-free echo. We are running Qwen3-VL-32B and a closed-model behavioral check (STI/SIT/STIT on a NaturalBench subset) and will include both in the revised paper. We agree that 1/8-resolution and reversed echoes keep coarse content. A stronger control we have run: randomly permuting the echoed copy's visual tokens matches the intact echo (Qwen3-VL-8B NaturalBench 37.95 vs 37.37, p = 0.25; Winoground 40.25 vs 40.25, p = 1.00; Gemma-3-27B p = 0.40 and 0.86). Token-matched content-free echoes (blank, noise, unrelated image at ~91 tokens) are running and will be added to Table 4, and we will scope the title's claim to what they support. We will also scope it to the image echo: echoing only the answer instruction instead of the question recovers little (29.21 vs 35.00), so on the text side the question's content is needed.

Q3. Broader cross-model results. Table 6 covers five families on NaturalBench, four of which show the paradox, and Table 7 covers three models on four image benchmarks. We are extending Table 7 to Qwen2.5-VL-7B and InternVL3-8B on the nine image benchmarks, with one alternative chat template per model (question and image in separate user turns) as the factorial the reviewer asks for, and will include it in the revised paper.

Q4. Probe selection bias. The reviewer is right that conditioning on STI-wrong / SIT-right guarantees a final-layer P(correct) gap; we will say so in the caption. The attention contrast and the layer where the orderings fork are not guaranteed by the selection, but could be amplified by it. We are rerunning the read-out probe and decision-layer analysis on (i) 300 random pairs, (ii) pairs both orderings get right, (iii) pairs both get wrong, and (iv) the reverse-disagreement set (STI right, SIT wrong), and will report answer-to-question attention at layer 23 for each in the revised paper. Prediction: lower answer-to-question attention under STI appears in all four sets, because it follows from the layout, while the late fork in P(correct) is specific to the disagreement set.

Q5. Scope of the GEPA result. We agree. GEPA cannot move the image, so the experiment shows what text-only optimization under a fixed layout recovers, not that no wording could. We will restate the conclusion accordingly. The result we rely on is the asymmetry: the optimizer gains under STI on all three benchmarks and returns the seed prompt unchanged under SIT on two of three, which a read-out account predicts and a wording account does not. We are also running a second optimizer (DSPy MIPRO) on NaturalBench and will include it in the revised paper.

Generality and Gemma. We agree the mechanism is established on Qwen3-VL-8B and does not explain Gemma; we will move this from the Limitations to the introduction. Gemma-3 also differs from Qwen3-VL in attention mask (bidirectional within image blocks), one candidate reason it lacks the paradox. The other candidate, the training mix, is tested by the balanced-order fine-tuning we describe in our response to Reviewer 4yeY, which we are running for the revised paper. Either way, the paradox is present in four of five families, large enough to exceed typical model gaps, and unreported in benchmark protocols, which is the practical point of the paper.

Reviewer_hYFTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper show the phenomenon which is called "question-first paradox" by the authors and investigate about it. "question-first paradox" means that placing the question before the image often leads to worse performance than placing it after the image.

The authors' prior work show this phenomenon on yes/no vision-language benchmarks, and this paper extends the investigation to a broader range of image and video QA tasks. The authors compare four prompt layouts: image-first (SIT), question-first (STI), question echoing (STIT), and image–question pair echoing (SITIT) . The experimental results demonstrate that STI underperforms SIT significantly, and SITIT achieves the best performance. STIT can recover most of the performance degradation caused by question-first prompting with a low additional cost. Further resolution experiments show that using a low-resolution second image in SITIT can still retain the benefits of repeating the full-resolution image. In contrast to the benefit on image benchmark, the benefits of image echoing are limited on video tasks.

Critical strengths
  1. The paper systematically reveals the "question-first paradox". Compared to the authors' previous work, this phenomenon is replicated across a broader range of image and video benchmarks. Additionally, the experiments of more VLMs on NaturalBench also show the generality of this phenomenon.

  2. This paper propose some methods to mitigate or repair this "question-first paradox" phenomenon. Question echoing (STIT) can recover most of the performance degradation with a low additional cost, while image–question pair echoing (SITIT) can further improve performance on many image benchmarks. The resolution experiments also show that using a low-resolution second image in SITIT can retain the benefits of the full-resolution echo while substantially reducing the additional token cost. These results provide useful guidance for balancing performance and input cost. Additionally, the reversal of the ordering effect on OD tasks is also an interesting finding. Unlike most visual question-answering tasks, OD benefits from placing the question before the image. This result shows that the effect of prompt ordering depends on the task types and provides a useful finding to understand the "question-first paradox" phenomenon.

Critical weaknesses
  1. The experiments regarding STITI is not sufficient, which is crucial because it is directly corresponding to the paper title "Position, Not Content". I think the the effects of position and content are not fully separated in the STITI experiment. In the paper's setting, the second image in SITIT is process by downsampling or reversing and it still preserves performance on some benchmarks. However, downsampling or reversing still retain visual information (i.e. Content stated in the paper title) from the original image. Therefore, these results don't clearly show that the improvement comes from position instead of the remaining visual content. Several additional controls such as replacing the second image with a blank or unrelated image or noisy image would help clarify authors' statement.

  2. The diagnostic tools (like logit-lens and attention analysis) are helpful, but they are mostly introduced from the authors’ prior work and are mainly summarized in this paper. This paper does not extend these diagnostics to the newly evaluated benchmarks and task types, particularly object detection in which the ordering effect reverses. Thus, beyond just running more evaluation, the paper doesn't really provide much new contribution.

Questions for authors
  1. [For weakness 1] To prove that position really matters more than content, authors could try to replace the second image in SITIT with a blank image, pure noise, or a completely unrelated picture.

  2. [For weakness 2] It could be considered to use diagnostic tools o the new benchmarks and task types.

1 comment

We thank the reviewer for the clear summary and for noting the detection reversal as a useful finding.

W1 / Q1. Separating position from content. We agree this is the key control for the title's claim, and that downsampling and reversal both keep visual content.

Already run: a random permutation of the echoed copy's visual tokens, which removes all spatial structure while keeping token count.

ModelBenchmark (group acc.)SITSITITpermuted echovs SITIT
Qwen3-VL-8BNaturalBench35.0537.3737.95p = 0.25
Qwen3-VL-8BWinoground31.7540.2540.25p = 1.00
Gemma-3-27BNaturalBench22.6325.5324.89p = 0.40
Gemma-3-27BWinoground20.7531.7531.25p = 0.86

Running now, as the reviewer proposes: the echo replaced by a blank image, Gaussian noise, and an unrelated image from the same benchmark, each token-matched to the 1/8 echo. Our prediction, stated before the result: blank and noise recover most of the gap; the unrelated image is the hardest case because it supplies competing evidence. We will add these to Table 4 in the revised paper. If content-free echoes fail, we will retitle and restate the claim as "the echo needs very little of the image", which the resolution, reversal and permutation controls support.

One related result we will add: for the text echo, content matters. Echoing only the answer instruction ("Answer yes or no.") instead of the question gives NaturalBench 29.21, near STI (26.95) and far below the full question echo (35.00, p = 4.7 × 10⁻⁹). This fits the read-out account, since the answer position must have the question nearby to read it, and we will scope "position, not content" to the image echo.

W2 / Q2. Diagnostics on the new tasks. We agree this would make the new results more than breadth. We are running the read-out probe (answer-to-question and answer-to-image attention, layerwise P(correct)) and the patch-difference procedure on (i) RF20 presence vs RF20 detection, the same images with opposite signs, and (ii) CV-Bench, the largest new deficit (−18.01), and will include them in the revised paper. Detection has no single answer position, so we will measure attention from the first box-coordinate token. Prediction: on presence, STI shows the low answer-to-question attention of Figure 4; on detection the box tokens use the question-steered patches directly, so the deficit does not cost accuracy.

On contribution: the automated patch-difference procedure (Section 4.2), the resolution sweep and token accounting, the detection and video boundaries, and the GEPA asymmetry test are all new to this paper, not carried over from the prior work.

Reviewer_4yeYOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies how the ordering of questions and images in vision-language model prompts affects performance. It shows that placing the question before the image often causes a read-out failure: although the question influences visual representations, the answer position fails to attend to the question properly. Across 13 additional image and video benchmarks, question-first prompting reduces accuracy by 7.01 points on average, while repeating the question or using a low-resolution image echo largely repairs the damage at much lower cost. The effect is not universal, however: question-first prompting improves object detection, and image echoing is less effective for long video inputs or tasks involving small visual targets.

Critical strengths

Important research question: It identifies prompt layout as an overlooked confound in VLM evaluation, showing that merely changing question–image order can cause performance differences larger than those between models.

Strong mechanistic analysis: Through representation probing, attention analysis, decision-layer tracking, and attention knockout, it convincingly distinguishes a read-out failure from a visual-perception failure.

Carefully scoped validation: Experiments across image QA, video QA, object detection, multiple benchmarks, and several model families demonstrate both the generality of the effect and its important boundaries.

Actionable insight: The ablations show that image echoes mainly provide a better read-out surface rather than a second visual perception, leading to low-cost fixes and task-specific recommendations for modern VLM systems.

Critical weaknesses

The paper does not fully disentangle architectural effects from training-distribution effects. The question-first degradation may reflect a mismatch with the model’s instruction-tuning data or chat-template conventions, where image-before-question prompts were seen more frequently. Because the same phenomenon largely disappears in Gemma models, the evidence does not establish that it is a general consequence of causal attention rather than a model-specific formatting prior. A stronger test would train or evaluate models under deliberately balanced question–image orders.

The evidence on model scale and modern VLMs is still limited. Qwen3-VL-8B is used for nearly all mechanistic analyses, resolution ablations, cost measurements, and prompt-optimization experiments. Although the paper includes several other models, the larger-model evidence is comparatively narrow, with no systematic scaling study across model sizes or architectures. Therefore, it remains unclear whether the effect is mainly a weakness of smaller models or persists in substantially larger and more capable VLMs.

The benchmark coverage is broad in number but narrow in task structure. Most experiments concern short-answer visual question answering with a single answer read-out position; detection is the main exception. The RF20 results themselves show that output format can reverse the effect, suggesting that the finding may be specific to token-level answer generation rather than multimodal reasoning in general. The headline 7.01-point average is also an unweighted benchmark-level mean over datasets with highly different sample sizes, so it should not be interpreted as a universal item-level performance loss.

The mechanistic evidence is persuasive but not yet conclusive. Logit-lens decodes and attention weights are useful diagnostics, but they do not by themselves prove that the proposed read-out pathway is the fundamental cause. The knockout experiment is restricted to attention from the answer position and to a selected NaturalBench disagreement set; the authors themselves report that broader query-side interventions fail to reproduce part of the dissociation. More decisive evidence would require activation patching, head- or path-level mediation, and truly uninformative echo controls such as randomized or feature-space-corrupted images.

The practical recommendations are not fully validated, and the presentation sometimes overstates the evidence. The reported “cost” of image echoing counts additional tokens but does not measure end-to-end latency, vision-encoder computation, memory, or throughput in realistic multi-image and video systems. Likewise, GEPA tests only one constrained form of text-only optimization, so its failure to match echoing does not prove that no alternative prompt engineering strategy could do so. Finally, the idiosyncratic terminology, limited related-work positioning, and separation between the prior-paper recap and new results make the central claims and experimental assumptions harder to audit.

Questions for authors

How can you determine whether the question-first degradation is a fundamental read-out problem rather than a model-specific bias from instruction tuning or prompt formats, especially since Gemma models do not show the same effect and most mechanistic analysis uses Qwen3-VL-8B?

Why were the causal interventions restricted mainly to the answer position and a selected disagreement set, and can you provide stronger controls—such as randomized or corrupted image echoes and end-to-end latency/computation measurements—to validate the proposed low-resolution echo strategy?

2 comments

We thank the reviewer for a detailed review and for finding that the probes and knockout separate a read-out failure from a perception failure. We address the soundness concerns in turn.

W1 / Q1. Architecture vs. instruction-tuning prior. We agree that the paper does not separate architecture from training distribution, and we will state this in the abstract and introduction, not only in the Limitations. We do not claim that causal attention forces the paradox. Our claim is narrower: on the open VLMs practitioners deploy, the layout intuition favors costs up to 18 points, and the failure sits in read-out. That holds whatever its origin, and agentic pipelines that prepend a goal and append retrieved images produce this layout by construction.

Still, a pure "unfamiliar template" prior does not fit two results already in the paper:

  1. The sign depends on the task, not the template. On RF20 the same model sees the same images under the same question-first layout. Asked a yes/no presence question, question-first hurts on 7 of 7 categories (−11.36, Table 5). Asked for boxes, it is better by 2.80 mAP and wins on 17 of 20 datasets (Table 3). A prior against question-first templates predicts harm in both.
  2. A rarer template repairs it. STIT (question, image, question) is less common in instruction data than STI, yet its 13 extra tokens recover 78% of the deficit across 13 benchmarks. A format prior predicts that moving further from the training template hurts more.

Gemma does not isolate the two factors either. Both Gemma models we test (Gemma-3-27B, Gemma-4-31B) lack the paradox, but Gemma-3 also differs architecturally: it applies a bidirectional attention mask within image blocks, while Qwen3-VL is causal over image tokens. We are running a comparison across Gemma generations (Gemma-3, Gemma-4 and the earlier Gemma-based PaliGemma-2) to test whether the absence of the paradox is stable across training recipes within one family, and will include it in the revised paper.

The balanced-order test the reviewer proposes is the right experiment, and we are running it: we LoRA fine-tune Qwen3-VL-8B on an instruction mix with question-before-image and question-after-image balanced, and will report the STI-SIT gap before and after on NaturalBench and Winoground in the revised paper. If the gap closes, the paradox is a trainable prior and echoing is the inference-time fix for models already shipped; if it persists, the cause lies in the architecture.

W2. Scale and model coverage. The paradox appears in 4 of 5 families on NaturalBench (Table 6) and is often larger outside Qwen3-VL-8B: Qwen2.5-VL-7B loses 17.6 group-accuracy points, LLaVA-1.5 12.3, InternVL3-8B 7.9. We acknowledge that the largest model we test, Gemma-4-31B, does not show it, so we cannot rule out that scale reduces it. We are running Qwen3-VL-32B on NaturalBench, Winoground and four image benchmarks as a within-family scale series, and STI/SIT/STIT on a closed model over a NaturalBench subset, and will include both in the revised paper.

W3. Task structure and the unweighted mean. The 13 benchmarks are not all yes/no: they include multiple choice (BLINK, CV-Bench, MMStar, MVBench, NExT-QA), counting (TallyQA) and open-ended short answers (VQAv2, MSVD-QA, TGIF-QA). They do share a short answer read out at the end, which is the scope the detection reversal identifies, and we will scope the title, abstract and Section 6 to short-answer read-out.

The reviewer is right that 7.01 is an unweighted benchmark mean. We will report it alongside: the median benchmark delta (−8.19), the item-weighted mean (−3.59, with 69% of the weight on VQAv2 at −2.94), and the item-weighted mean without VQAv2 (−5.02). Item-weighted, the image-echo gain is −0.21, driven by VQAv2 and the three large video sets, which supports our recommendation to echo the question on video.

W4 / Q2. Strength of the causal evidence. We agree that logit lens and attention mass are diagnostics; causal claims rest on the knockout. We restrict the query side to the answer position because that is where the hypothesized read-out happens. Widening it also severs the image tokens' own attention, and the random-span control then moves STI by 10 points, so that configuration cannot separate paths (Appendix J). We will add:

  • Corrupted echoes. A random permutation of the echoed copy's visual tokens (patch features and positions shuffled together), which destroys spatial structure, matches the intact echo: Qwen3-VL-8B NaturalBench group accuracy 37.95 vs 37.37 (p = 0.25), Winoground 40.25 vs 40.25 (p = 1.00); Gemma-3-27B p = 0.40 and 0.86. It still beats question-last (Qwen NaturalBench p = 1.5 × 10⁻⁴). We are also running blank, Gaussian-noise and unrelated-image echoes, token-matched to the 1/8 echo, and will add them to Table 4.
  • Activation patching (running, for the revised paper): we patch SIT's answer-position residual stream into the STI run layer by layer and report recovery of P(correct), testing sufficiency where the knockout tests necessity.
  • Knockout beyond the disagreement set (running, for the revised paper): the same knockout on a random 250-pair sample and on pairs where both orderings agree.

W5. Cost, GEPA, presentation. (a) We are measuring prefill latency, peak memory and throughput per ordering for Qwen3-VL-8B, for single images and six-frame video, and will add them next to the token counts in Table 11. Vision-encoder cost also scales with pixels, so the 1/8 echo adds about 1/64 of an encoder pass. (b) We agree GEPA tests one text-only optimizer under a fixed layout. We will restate the conclusion as "a text-only optimizer that cannot move the image recovers 26% to 83% of the gap and finds nothing under SIT" and remove "rewording cannot replace it". (c) We will add a notation glossary, expand related work on VLM prompt sensitivity, and merge the recap with the new results so that each claim and its evidence sit together.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.