- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies a fairly simple but practically relevant prompt-formatting issue for VLMs: whether the question is placed before or after the image. The main observation is that, for several autoregressive VLMs, putting the question first can noticeably hurt accuracy even though the model receives exactly the same image and text. The proposed fix is also simple: either repeat the question after the image, or repeat the image after the question.
Critical strengths
-
The paper provides strong controlled evidence for the prompt-layout effect. The central STI-versus-SIT comparisons preserve prompt content and token count while changing order, and the paper evaluates the effect over many datasets and output formats. This is an important question since unreported layout differences can exceed the model gaps benchmark papers aim to measure.
-
The paper offers comprehensive analysis on this problem. The author actively probes boundaries, for example, detection reverses the ordering effect, image echoing is weak on video, POPE does not benefit, and Gemma does not show the same paradox. Reporting these limits, plus exact token costs and the GEPA comparison, makes the work more useful.
Critical weaknesses
-
The novelty is constrained by substantial overlap with the authors’ prior workshop paper. According to the submission, the prior paper already established the paradox on three benchmarks, the read-out diagnosis, attention knockout, both echo repairs, and the reversed-image control. The new contributions are breadth, detection/video boundaries, resolution/cost analysis, automated patch visualization, and a prompt-optimizer comparison. These are useful, but the submission’s main conceptual story and intervention are not new to this paper.
-
The claimed “position, not content” mechanism is overgeneralized. The causal analysis is performed primarily on Qwen3-VL-8B and explicitly does not explain the Gemma results, where image echoing helps without the paradox. This also raises the possibility that other VLMs, including stronger closed-source models that are not evaluated in the paper, may not exhibit the paradox at all. Furthermore, a 1/8-resolution echo still adds about 91 image tokens (and 107 total tokens) and may preserve coarse scene/object information, and spatial reversal also preserves most semantic content. Neither control is equivalent to content-free tokens.
-
The new image/video benchmarks are reported primarily for Qwen3-VL-8B. Cross-model evidence is concentrated on NaturalBench and four selected image benchmarks, while the detailed mechanistic study is almost entirely Qwen-specific. Since Gemma does not exhibit the paradox, a broader factorial evaluation across architectures and prompt templates is necessary.
-
The read-out probe uses cases where STI is wrong and SIT is right, a selection that necessarily produces large final-layer probability differences and may amplify attention contrasts. This subset is appropriate for studying failures, but the paper should also report the same measurements on an unconditioned sample, the reverse-disagreement set, and cases where both orderings agree to establish whether the signature is specific to the proposed failure mode.
-
The GEPA experiment is narrow. It uses a single optimizer, a separate reflection model, small train/validation sets, and only three benchmarks. GEPA is also prohibited from changing layout, so the result mainly shows that text-only editing cannot always compensate for a positional constraint, not that “better wording” could not recover the gap.
Questions for authors
- Could the authors clearly distinguish the contributions of this submission from the prior workshop paper, including which claims and experiments are genuinely new?
- Does the paradox occur in stronger or closed-source VLMs? Have the authors tested a content-free, token-matched echo to isolate position from residual visual information?
- Can the authors provide broader cross-model results beyond NaturalBench and the four selected image benchmarks, especially given that Gemma does not exhibit the paradox?
- Do the read-out and attention patterns remain on unconditioned samples, reverse-disagreement cases, and examples where both orderings agree?
- Given that GEPA cannot modify prompt layout, what evidence supports the broad claim that better wording cannot recover the gap rather than a conclusion specific to GEPA?
2 comments
Q4. Probe selection bias. The reviewer is right that conditioning on STI-wrong / SIT-right guarantees a final-layer P(correct) gap; we will say so in the caption. The attention contrast and the layer where the orderings fork are not guaranteed by the selection, but could be amplified by it. We are rerunning the read-out probe and decision-layer analysis on (i) 300 random pairs, (ii) pairs both orderings get right, (iii) pairs both get wrong, and (iv) the reverse-disagreement set (STI right, SIT wrong), and will report answer-to-question attention at layer 23 for each in the revised paper. Prediction: lower answer-to-question attention under STI appears in all four sets, because it follows from the layout, while the late fork in P(correct) is specific to the disagreement set.
Q5. Scope of the GEPA result. We agree. GEPA cannot move the image, so the experiment shows what text-only optimization under a fixed layout recovers, not that no wording could. We will restate the conclusion accordingly. The result we rely on is the asymmetry: the optimizer gains under STI on all three benchmarks and returns the seed prompt unchanged under SIT on two of three, which a read-out account predicts and a wording account does not. We are also running a second optimizer (DSPy MIPRO) on NaturalBench and will include it in the revised paper.
Generality and Gemma. We agree the mechanism is established on Qwen3-VL-8B and does not explain Gemma; we will move this from the Limitations to the introduction. Gemma-3 also differs from Qwen3-VL in attention mask (bidirectional within image blocks), one candidate reason it lacks the paradox. The other candidate, the training mix, is tested by the balanced-order fine-tuning we describe in our response to Reviewer 4yeY, which we are running for the revised paper. Either way, the paradox is present in four of five families, large enough to exceed typical model gaps, and unreported in benchmark protocols, which is the practical point of the paper.
We thank the reviewer for precise questions and for recognizing the controlled design and the boundaries we report.
Q1. What is new relative to the workshop paper.
The workshop paper asks whether the paradox exists and why. This paper asks how far it reaches, what the repair is made of, and what it costs. We will shorten the recap so the new results take most of the main text.
Q2. Stronger models; content-free echo. We are running Qwen3-VL-32B and a closed-model behavioral check (STI/SIT/STIT on a NaturalBench subset) and will include both in the revised paper. We agree that 1/8-resolution and reversed echoes keep coarse content. A stronger control we have run: randomly permuting the echoed copy's visual tokens matches the intact echo (Qwen3-VL-8B NaturalBench 37.95 vs 37.37, p = 0.25; Winoground 40.25 vs 40.25, p = 1.00; Gemma-3-27B p = 0.40 and 0.86). Token-matched content-free echoes (blank, noise, unrelated image at ~91 tokens) are running and will be added to Table 4, and we will scope the title's claim to what they support. We will also scope it to the image echo: echoing only the answer instruction instead of the question recovers little (29.21 vs 35.00), so on the text side the question's content is needed.
Q3. Broader cross-model results. Table 6 covers five families on NaturalBench, four of which show the paradox, and Table 7 covers three models on four image benchmarks. We are extending Table 7 to Qwen2.5-VL-7B and InternVL3-8B on the nine image benchmarks, with one alternative chat template per model (question and image in separate user turns) as the factorial the reviewer asks for, and will include it in the revised paper.