acceptodds
Under review as a conference paper at ICLR 2027

The Answer Format Is Part of the Measurement: Option Labels Change Visual Context Effects in VLMs

Abstract

Vision-language models (VLMs) are increasingly evaluated for how much they rely on cultural and regional context. A common protocol holds a target fixed, changes its visual surroundings, and compares the answers the model chooses among labeled options. However, we argue that such measurements depend on how the options are labeled. We introduce Scene-Orthography, a controlled benchmark built on a cultural convention with an unambiguous answer: a model must identify which spelling of a word, American or British (e.g., COLOR vs. COLOUR), appears on a sign whose pixels are identical in a US and a UK street scene, so the correct answer never depends on the scene. Holding the images fixed, we vary only the answer format: whether the options are labeled with country names or with letters that carry no regional meaning, which spelling each label names, whether the spellings are shown directly, and where the labels are placed. Across eight open models, country-name labels enlarge the measured effect of the scene, and in the two primary models, swapping which spelling each label names reverses its direction. A pre-specified test on held-out words and photographs shows that labels act through the option they sit on: a label beside an option steers the choice even when the prompt declares it unassigned, whereas the same words on a separate line have little effect. This dependence varies widely across models and shrinks with scale within one model family, and on the natural-image benchmark MMVP, the answer interface also changes measured accuracy. Therefore, a visual context effect is a property of both an image and an answer format, suggesting that evaluations of cultural context should report both.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
vision-language models, multimodal evaluation, visual context, answer format sensitivity, multiple-choice question answering

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_NStUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper investigates whether the measured influence of visual context on VLM predictions depends on how answer options are presented. The authors introduce Scene-Orthography, a controlled benchmark for evaluating the interaction between visual context and answer formats. Through a series of controlled experiments across multiple VLMs, they demonstrate that option labels, their semantic associations, and their placement can substantially alter the measured context effect, even reversing its direction. The work primarily provides an empirical characterization of this phenomenon, without establishing its underlying mechanisms or generalizability beyond the studied settings.

Critical strengths
  1. An interesting and relevant research question. The paper investigates whether measured visual context effects in VLMs are influenced by answer formatting. This is an interesting question with practical implications for the reliability and interpretation of VLM evaluations.

  2. Careful experimental design. The authors conduct a series of controlled experiments to isolate the effects of option labels, their mappings, and their placement. The use of held-out validation and multiple VLMs provides relatively convincing empirical evidence for the observed phenomenon.

Critical weaknesses
  1. Limited novelty and depth of insights. The paper largely remains at the level of empirical observations. The sensitivity of LLMs and VLMs to option ordering, labeling, and prompt formatting has been extensively studied [1, 2]. Prior work has also investigated the causes of such biases, including token-level preferences [3] and the causal roles of attention heads [4]. While this paper identifies a specific interaction between visual context and answer labels, it provides limited new understanding beyond these established findings. In particular, the proposed explanation remains behavioral, without offering deeper insights into the underlying mechanisms.

  2. Limited conceptual development and generalizability. Despite the large number of experiments, the paper reads more like an extensive experimental report than a study that develops substantial new insights. Most experiments involve narrowly scoped variations of answer formats, repeatedly examining the same phenomenon without advancing toward a more general understanding. Moreover, the central findings are confined to a highly specific US/UK spelling task. The attempted transfer experiments on CLEVR and VQAv2 in Appendix F did not provide evidence of generalization, while the MMVP experiments only demonstrate answer-format sensitivity in accuracy rather than the proposed visual-context interaction. These limitations significantly weaken the broader contribution of the work.

[1] Fool Your (Vision and) Language Model with Embarrassingly Simple Permutations. ICML 2024.

[2] Benchmarking and Mitigating MCQA Selection Bias of Large Vision-Language Models. EMNLP 2025.

[3] Large Language Models Are Not Robust Multiple Choice Selectors. ICLR 2024.

[4] Option Symbol Matters: Investigating and Mitigating Multiple-Choice Option Symbol Bias of Large Language Models. NAACL 2025.

Questions for authors

None

2 comments
AuthorsOpenReview ↗

Response to Reviewer NStU (1/1)

Dear Reviewer NStU,

Thank you for recognizing our research question as relevant and our experimental design as careful. In the revised PDF, changes for your review are green and changes shared with other reviewers are purple, each tagged with your weakness W1 or W2.


W1. The novelty and depth are limited given [1]–[4], and the explanation remains behavioral.

A1: Thank you for this point. Our effect follows a route that the mechanisms in [1]–[4] cannot produce, and a new intervention locates part of it:

  • Known mechanisms. The sensitivity documented in [1, 2] and traced to token-level priors [3] and attention heads [4] does not depend on the image, and our regional and neutral formats share the same option letters and rotations, so these mechanisms cannot by themselves produce the label-dependent scene effects we measure.
  • Internal intervention. In Qwen3-VL-8B, patching the label-token states between scenes transfers 6.0 pp more of the scene effect than a matched instruction-token patch, a reliable difference.

We added the intervention at the end of Section 5.2, with details and an attention-masking test in the new Appendix C.5, and related our work to [3, 4] in Section 2. Therefore, the scene reaches the answer partly through the label tokens on the option lines, a route distinct from the image-independent mechanisms in [1]–[4].


W2. The paper reads like an experimental report, and its findings are confined to a spelling task.

A2: Thank you for this comment. We reorganized the paper around one proposition, and we show a consequence beyond the spelling task:

  • One proposition. Each study isolates one component of the answer format, and together they support one claim: labels that name the manipulated dimension and sit on the option lines add a component to the context-effect estimate. Its predictions hold: swapping what the labels name reverses the effect, and moving labels off the option lines removes it. In a new crossed test, aligned country labels produce 11.3 and 12.2 pp, whereas meaningful but non-aligned time-of-day labels produce 1.1 and 2.9 pp, with a positive pre-specified alignment contrast, although the full joint criterion is not met.
  • An existing benchmark. On WorldCuisines, with images and answer contents fixed, a key placed after the choices lowers the rank correlation with direct presentation to 0.286, compared with 0.929 for a key before the choices, and reverses six model pairs reliably; under greedy generation, it reliably lowers accuracy in all eight models.

We stated the proposition in Section 1, added the crossed test as the new Section 5.3, added the WorldCuisines results as a new paragraph and Table 2 in Section 7 with details in Appendices E.3 and E.4, and moved MMVP to Appendix E.2; Appendix F reports our first attempts on CLEVR and VQAv2. Therefore, the experiments form one testable account, and its consequence extends to an existing cultural benchmark.


Thank you again for your feedback, which has strengthened the manuscript. We hope our response addresses your concerns, and we kindly ask you to reconsider your evaluation.

Best,

Authors

AuthorsOpenReview ↗

Gentle Reminder

Dear Reviewer NStU,

Thank you once again for your thoughtful review and the time you have devoted to our paper. As the discussion period is drawing to a close, we wanted to kindly follow up on our responses. We hope they have addressed your concerns. If anything remains unclear, we would be glad to provide further clarification promptly.

If you feel our responses have resolved your concerns, we would sincerely appreciate it if you could take them into account in your final evaluation.

Best regards,

The Authors

Reviewer_VGTnOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This work asked whether answer format would impact VLM multiple-choice VQAs, given how the visual input would impact the results. The work introduced a method called Scene-Orthography, which is controlled benchmark where textual targets are put into street scenes in the U.S. and U.K. And the authors change semantic content, assignment, placements in the experiments, and they show that regional labels could impact the magnitude and direction of scene effect. So the central conclusion is that visual context impacts depend both on the image, which is the visual context, but also the answer interface.

Critical strengths

This is a very legit method question in VLM evals: basically, does the answer format would impact the model behaviors? The counterfactual design by the authors, which they preserve the target pixels but change the surrounding areas, is a interesting way to probing this question.

Critical weaknesses

I am quite concerned about the novelty of this paper. I have read a lot of works in this domain, basically how to eval VLM VQAs, and I don't find the authors have found something particularly new. For examples - some of the works include: Zong et al. (ICML 2024), Sclar et al. (ICLR 2024), Xue et al. (ACL 2024), Atabuzzaman et al. (EMNLP 2025), and Zhang et al. (2025) . While the authors argue that their contribution is more on demonstrating how answer format change the estimated effects of visual intervention, not just accuracy. But It seems to be trivial and obvious and do not constitute a new phenomenon. But also, they did not demonstrate how the work and experimental design would generalize beyond their benchmark.

Questions for authors

What concrete prediction of the central scene-by-label interaction would not follow from previously established semantic answer-option biases? Can the authors articulate the genuinely new theoretical claim more precisely?

2 comments
AuthorsOpenReview ↗

Response to Reviewer VGTn (1/1)

Dear Reviewer VGTn,

Thank you for recognizing that our paper addresses a legitimate methodological issue and that our counterfactual design keeps the correct answer fixed. In the revised PDF, changes for your review are vermilion and changes shared with other reviewers are purple, each tagged with your weakness W1 or question Q1. We also reduced the main-text notation to three quantities, τ, J, and K, to make the paper easier to follow.


W1 and Q1. The novelty appears limited given known option biases. What is the new theoretical claim?

A1: Thank you for this question. Our claim is a testable proposition: when option lines carry labels that name the dimension a context manipulation varies, the context-effect estimate includes a component set by what those labels denote and where they sit. This claim differs from the works you cite:

  • Selection bias and symbol binding. Position and symbol preferences cannot produce the effect, because our regional and neutral formats share the same option letters and rotations.
  • Prompt-format sensitivity. Prior work measures changes in accuracy, whereas we measure the change in a scene contrast; accuracy can even rise while this contrast grows.
  • Vision–option shortcuts. Image–option matching is the likely mechanism, and we show its consequences for measurement: swapping only which spelling each country names reverses the estimated effect, from +19.7 to −47.4 pp in Qwen3-VL-8B, and standard safeguards, such as option rotation and an instruction declaring labels unassigned, do not remove the effect.

We stated the proposition in the sixth paragraph of Section 1 and added the paragraph "Standard safeguards do not remove the effect" in Section 7. Therefore, our claim is a measurement claim: a known mechanism makes context-effect estimates depend on the answer interface in ways that existing checks do not reveal.


Q1. What concrete prediction would not follow from established semantic option biases?

A2: Thank you for this question. Established accounts say that a model matches the image against option content, but they do not specify where a label must act or whether its meaning alone is enough; our proposition predicts both, and both predictions hold:

  • Placement, not stated meaning. If region names acted by raising the salience of the region anywhere in the prompt, they would matter wherever they appear. In a pre-specified held-out test, region names beside the options add 7.3 and 10.3 pp to the scene effect, whereas the same names on a separate line add almost nothing, and declaring them unassigned while they stay beside the options leaves the effect unchanged.
  • Alignment, not meaning alone. In a new crossed test, country labels produce attachment effects of 11.3 and 12.2 pp, whereas meaningful but non-aligned time-of-day labels produce 1.1 and 2.9 pp, and the pre-specified alignment contrast is positive in both models. The full joint criterion is not met, because Qwen3-VL-8B read the day/night signs perfectly and the off-diagonal equivalence bounds were not met.

The alignment that the review describes as deliberate is therefore the condition that our proposition identifies, not an incidental feature of the task: without it, meaningful labels add little. We added the crossed test as the new Section 5.3, "Alignment, Not Meaning Alone," with every cell in Appendix F. Therefore, the interaction follows predictions about where labels act and when they matter that established option-bias accounts do not make.


W1. There is little evidence that the interaction generalizes beyond the spelling task.

A3: Thank you for this point. The answer interface changes model rankings on the existing cultural benchmark WorldCuisines. With images, questions, and answer contents fixed, a key before the choices leaves the ranking nearly unchanged, with a Kendall rank correlation of 0.929, whereas a key after the choices lowers it to 0.286 and reverses six model pairs reliably under the likelihood readout; under greedy generation, the same key reliably lowers accuracy in all eight models. We added these results as a new paragraph and Table 2 in Section 7, with details in the new Appendices E.3 and E.4, and Appendix F reports our first attempts on CLEVR and VQAv2. Therefore, the dependence on the answer interface extends to an existing benchmark, and transfer to non-reading tasks is the next step.


Thank you again for your feedback, which has strengthened the manuscript. We hope our response addresses your concerns, and we kindly ask you to reconsider your evaluation.

Best,

Authors

AuthorsOpenReview ↗

Gentle Reminder

Dear Reviewer VGTn,

Thank you once again for your thoughtful review and the time you have devoted to our paper. As the discussion period is drawing to a close, we wanted to kindly follow up on our responses. We hope they have addressed your concerns. If anything remains unclear, we would be glad to provide further clarification promptly.

If you feel our responses have resolved your concerns, we would sincerely appreciate it if you could take them into account in your final evaluation.

Best regards,

The Authors

Reviewer_8JqvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper argues that the measured effect of visual context in VLMs depends on how the answer options are labeled. The authors build Scene-Orthography, where a sign with an American or British spelling is pasted with identical pixels into a US or a UK street photo, so the correct answer never depends on the scene. Across eight open VLMs, labeling the options with country names makes the scene matter more, swapping which spelling each country names reverses the effect, and a pre-specified test shows that labels only matter when they sit on the option lines. The effect shrinks with scale in Qwen3-VL, and an indirect answer key also lowers accuracy on MMVP.

Critical strengths
  1. The design is clean and careful. Pasting the same sign pixels into both scenes keeps the correct answer fixed, the format factors are changed one at a time, and the main attachment test is pre-specified and run on held-out words and photos.

  2. The practical advice is simple and useful: report the answer format, and check one neutral or detached format on the same images.

Critical weaknesses
  1. The main finding is close to expected. If a US street scene is shown and one option says "United States", it is not surprising that a model that cannot read the sign well picks that option, and that swapping what the label points to flips the effect. The sensitivity of multiple-choice answers to option labels and symbols is already well documented, including for VLMs (Zheng et al., 2024; Robinson & Wingate, 2023; Zong et al., 2024). The new part is measuring this on a context effect instead of accuracy, which is a modest step.

  2. The setting is narrow and its relevance to real evaluations is not shown. The task is synthetic and built so that the labels name exactly the dimension the scene changes. The paper does not show that any existing cultural or regional benchmark uses such a format or that its conclusions would change. The transfer attempts to CLEVR and VQAv2 did not pass the authors' own criteria, the MMVP result is about answer keys lowering accuracy, which is a different and known issue, and the effect almost disappears at Qwen3-VL-32B. Moreover, the released code covers prompt construction and analysis only. The stimuli, the sign renderer, the model adapters and the raw outputs are not provided, which limits the credibility of the reported results.

Minor issues:

  • The released code uses different prompt wording across studies ("candidate" in the attachment and factorial studies, "reference entry" and "Entry X/Y" elsewhere), while Appendix A says all formats share one template.
  • The many symbols (τ, J, K, E, B, R, ρ, Λ) make the results hard to follow.
  • Choices are read only from the log-probabilities of the letter tokens, so it is unclear whether the effect holds for generated answers.
Questions for authors

See Weaknesses. Also,

  1. Can you point to an existing cultural or regional VLM benchmark whose options share the dimension of its context manipulation, and show how much its reported effects change under neutral or detached labels?
  2. Six of the eight models failed the Binding control checks, and GLM-4.6V-Flash often misses the right spelling even when the target is given as text. How large is the scene effect if you keep only models and items that pass the HIGH and GOLD controls?
5 comments
AuthorsOpenReview ↗

Response to Reviewer 8Jqv (1/2)

Dear Reviewer 8Jqv,

Thank you for recognizing our clean design, the pre-specified held-out test, and our practical advice. In the revised PDF, changes for your review are blue and changes shared with other reviewers are purple, each tagged with your point: weaknesses W1–W2, minor issues M1–M3, and questions Q1–Q2. We also uploaded an expanded supplementary package.


W1. The finding is close to expected, and option-label sensitivity is already well documented.

A1. Thank you for this point. We agree that image–option matching is the likely mechanism; our contribution is to show what it does to a context-effect estimate. Position and symbol preferences cannot explain the effect, because our regional and neutral formats share the same option letters and rotations. Three consequences matter for evaluation:

  • The sign can reverse. Swapping only which spelling each country names changes the signed effect from +19.7 to −47.4 pp in Qwen3-VL-8B, as Section 4.2 of the original submission reports.
  • Safeguards fail. The effect survives option rotation and an instruction declaring the labels unassigned, as Section 5.2 of the original submission reports.
  • Accuracy checks miss it. Regional labels can raise accuracy while increasing the scene effect.

We added this point as a new paragraph in Section 7, “Standard safeguards do not remove the effect.” Therefore, existing option-bias checks do not reveal this format dependence, and evaluations need to test for it directly.


W2 and Q1. The relevance to real evaluations is not shown. Can we show an existing cultural benchmark whose conclusions change?

A2: Thank you for this question. The answer interface changes model rankings on the cultural benchmark WorldCuisines. With images, questions, and answer contents fixed for 499 questions and eight models, we replaced the answer text with neutral names plus a key placed before or after the choices:

  • Likelihood readout. A key before the choices leaves the ranking nearly unchanged, with a Kendall rank correlation of 0.929, whereas a key after the choices lowers it to 0.286 and reverses six model pairs reliably. Without GLM-4.6V-Flash, the correlation still falls from 0.905 to 0.333.
  • Generation readout. A key after the choices reliably lowers accuracy in all eight models. For the two Qwen3-VL models, whose invalid-output rates are at most 0.04%, the drops of 25.0 and 18.8 pp match the likelihood readout; for other models, the effect depends partly on format compliance.

We added these results as a new paragraph and Table 2 in Section 7, with details in the new Appendices E.3 and E.4, and moved MMVP to Appendix E.2. Therefore, the answer interface changes the conclusions of an existing cultural benchmark, not only accuracy on our task.


W2 and M3 The effect nearly disappears at Qwen3-VL-32B, and choices are read only from log-probabilities.

A3: Thank you for these points. We find that the effect holds for generated answers, and that the near-zero effect of Qwen3-VL-32B reflects the original evidence levels, at which the model is at ceiling. We added a reduced-legibility condition, chosen before we inspected any scene effect, which lowers the Plain accuracy of Qwen3-VL-32B to about 74%. The attachment effects under the two readouts, in pp, are:

ModelHIGH: likelihood / generationReduced: likelihood / generation
Qwen3-VL-8B7.42 / 7.1311.28 / 11.33
Qwen3-VL-32B0.00 / −0.107.52 / 7.62
GLM-4.6V-Flash9.28 / 7.0812.74 / 15.82

The two readouts closely match in every model, and off the ceiling the point estimate of Qwen3-VL-32B rises from near zero to 7.5 pp. We added these results to Sections 3.4 and 6.2, with details in the new Appendix D.2, and replaced the paragraph "Why larger models show less" in Section 8 with "Scale and evidence." Therefore, the effect on our task does not depend on the log-probability readout, and the scale trend holds at the original evidence levels rather than in general.

AuthorsOpenReview ↗

Response to Reviewer 8Jqv (2/2)

Q2. How large is the effect if we keep only items that pass the HIGH and GOLD controls?

A4: Thank you for this question. The Binding effect stays reliably positive under the strictest filter. We kept only words that each primary model reads correctly in every HIGH and GOLD control cell, with the same words in both scenes:

Study and effectQwen3-VL-8B wordsQwen3-VL-8B effect, ppGLM-4.6V-Flash wordsGLM-4.6V-Flash effect, pp
Binding, B34/4830.8735/4823.60
Naming, J16/645.6653/6412.68
Direct, J_D6/4830.990/48—
Attachment, K5/164.690/16—

The Binding effect in both models, the Naming effect in GLM-4.6V-Flash, and the direct-presentation effect in Qwen3-VL-8B remain reliably positive. The attachment test retains too few words, so its pre-specified estimate, which passed its own controls, remains primary. GLM-4.6V-Flash's lower GOLD accuracy comes mainly from choosing Neither rather than the other spelling. We added this analysis at the end of the first paragraph of Section 4.2, with every study in the new Appendix B.4. Therefore, the label-following effect does not depend on words that the models cannot read.


M1 and M2. The prompt wording differs across studies, and the many symbols are hard to follow.

A5: Thank you for catching these. We corrected the "one template" statement in Section 3.2 and Appendix A, and the new Table A2 gives the exact wording of every study, including the "Entry X" labels in Naming. Within each study, all formats share one question line and paired scenes receive identical prompts, which we confirmed on all 428,032 scene records. We also reduced the main text to three quantities, τ, J, and K, and added a factor of one half that was missing from the written definition of the signed effect, with all values unchanged. Therefore, every scene contrast compares identical prompts, and the main text is easier to follow.


W2. The released code covers prompt construction and analysis only.

A6: Thank you for this point. The revised supplementary package also includes raw per-presentation scores for the included experimental panels, the sign renderer, archived model-adapter source code, and CPU reproduction scripts and tests. These recompute the primary Naming, Binding, and Direct point estimates, the eight-model attachment point estimates, the primary attachment intervals, and the WorldCuisines accuracy and ranking analyses from the archived scores. We updated the Reproducibility Statement to list the same contents. Therefore, the main reported results can now be checked from the archived scores.


Thank you again for your feedback, which has strengthened the manuscript. We hope our response addresses your concerns, and we kindly ask you to reconsider your evaluation.

Best,

Authors

AuthorsOpenReview ↗

Gentle Reminder

Dear Reviewer 8Jqv,

Thank you once again for your thoughtful review and the time you have devoted to our paper. As the discussion period is drawing to a close, we wanted to kindly follow up on our responses. We hope they have addressed your concerns. If anything remains unclear, we would be glad to provide further clarification promptly.

If you feel our responses have resolved your concerns, we would sincerely appreciate it if you could take them into account in your final evaluation.

Best regards,

The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary Response by Authors

Dear Reviewers, ACs, SACs, and PCs,

We sincerely thank all reviewers for their careful and constructive feedback, which has strengthened the positioning, empirical support, and clarity of our manuscript. We summarize the key points of the discussion below.


Recognized Strengths:

Reviewer 8Jqv finds our design clean and careful and describes our practical advice as simple and useful; after the rebuttal, Reviewer 8Jqv confirmed that the new analyses address the minor concerns and Q2 and raised the soundness score. Reviewer VGTn recognizes that our paper addresses a legitimate methodological issue and that the target-preserving counterfactual design is a useful way to examine context-dependent errors. Reviewer NStU finds the research question interesting and relevant, and the controlled experiments with held-out validation relatively convincing.


Main Contributions:

Our work shows that the estimate of a visual-context effect depends on the answer interface, not only accuracy. With sign pixels, correct answers, and scene pairs held fixed, changing only which spelling a country label names reverses the signed scene effect, from +19.7 to −47.4 pp in Qwen3-VL-8B and from +17.4 to −33.6 pp in GLM-4.6V-Flash (Figure 1a). A pre-specified test on held-out words and photographs locates this dependence in labels attached to the option lines, and a factorial follow-up shows that a label beside an option steers the choice even when the prompt declares it unassigned. Image–option matching is the likely mechanism; our contribution is to measure its consequence for context-effect estimates and to identify when it arises.


Main Revisions and Additional Evidence:

All revisions are in the uploaded PDF, with reviewer-specific colors and point-level annotations. The Summary of Revisions follows the references. The major changes are as follows:

  1. Robustness to reading ability (§4.2, Appendix B.4): Keeping only words that each primary model reads correctly in every HIGH and GOLD control cell, the label-following Binding effect remains at 30.9 pp in Qwen3-VL-8B and 23.6 pp in GLM-4.6V-Flash.

  2. Generated answers and scale (§§3.4, 6.2, Appendix D.2): Likelihood and greedy-generation readouts give closely matching attachment effects. When reduced legibility brings Qwen3-VL-32B off its accuracy ceiling, its point estimate rises from near zero to 7.5 pp, so we present the scale trend as describing the original evidence levels.

  3. Conditions and mechanism (§§5.2–5.3, Appendices C.5 and F): In a new crossed test, labels that name the manipulated dimension produce attachment effects of 11.3 and 12.2 pp, whereas meaningful but non-aligned labels produce 1.1 and 2.9 pp. Label-token patching in Qwen3-VL-8B transfers 6.0 pp more of the scene effect than a matched control.

  4. Relevance to an existing benchmark (§7, Appendices E.3–E.4): On 499 WorldCuisines questions across eight VLMs, placing the answer key after the choices lowers rank agreement with direct presentation from 0.929 to 0.286 and reliably reverses six model pairs under the likelihood readout. Under greedy generation, the same change lowers accuracy in all eight models; for the two Qwen3-VL models, with invalid-output rates at or below 0.04%, the drops of 25.0 and 18.8 pp match the likelihood readout.

  5. Positioning, clarity, and reproducibility (§§1–3, 7–9; supplement): We stated the central claim as one testable proposition, documented the exact prompt wording of every study, reduced the main-text notation to three quantities, added a practical recommendation on key placement and readout, and released raw scores, the sign renderer, model adapters, and CPU reproduction scripts.


The remaining concerns relate to novelty relative to image–option matching and to scope, rather than to the correctness of the findings. We agree that the WorldCuisines audit shows a format effect on model rankings rather than the scene-by-label interaction, and we will state this scope in Section 7 of the final version.

We believe the paper offers three contributions of value to the community. First, it shows that a standard protocol for measuring visual-context effects can yield opposite conclusions on the same images, and that the usual safeguards against option bias, option rotation and explicit disclaimers, do not prevent this. Second, it supports this result with a pre-specified, held-out design that all three reviewers commended, together with strict reading controls, two readouts, and a reproducibility package. Third, it turns the result into a simple diagnostic that any evaluation of visual or cultural context can apply: report the answer format, and check the estimate under one neutral or detached format on the same images. We hope these contributions will be taken into account in the final decision.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.