- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper investigates whether the measured influence of visual context on VLM predictions depends on how answer options are presented. The authors introduce Scene-Orthography, a controlled benchmark for evaluating the interaction between visual context and answer formats. Through a series of controlled experiments across multiple VLMs, they demonstrate that option labels, their semantic associations, and their placement can substantially alter the measured context effect, even reversing its direction. The work primarily provides an empirical characterization of this phenomenon, without establishing its underlying mechanisms or generalizability beyond the studied settings.
Critical strengths
-
An interesting and relevant research question. The paper investigates whether measured visual context effects in VLMs are influenced by answer formatting. This is an interesting question with practical implications for the reliability and interpretation of VLM evaluations.
-
Careful experimental design. The authors conduct a series of controlled experiments to isolate the effects of option labels, their mappings, and their placement. The use of held-out validation and multiple VLMs provides relatively convincing empirical evidence for the observed phenomenon.
Critical weaknesses
-
Limited novelty and depth of insights. The paper largely remains at the level of empirical observations. The sensitivity of LLMs and VLMs to option ordering, labeling, and prompt formatting has been extensively studied [1, 2]. Prior work has also investigated the causes of such biases, including token-level preferences [3] and the causal roles of attention heads [4]. While this paper identifies a specific interaction between visual context and answer labels, it provides limited new understanding beyond these established findings. In particular, the proposed explanation remains behavioral, without offering deeper insights into the underlying mechanisms.
-
Limited conceptual development and generalizability. Despite the large number of experiments, the paper reads more like an extensive experimental report than a study that develops substantial new insights. Most experiments involve narrowly scoped variations of answer formats, repeatedly examining the same phenomenon without advancing toward a more general understanding. Moreover, the central findings are confined to a highly specific US/UK spelling task. The attempted transfer experiments on CLEVR and VQAv2 in Appendix F did not provide evidence of generalization, while the MMVP experiments only demonstrate answer-format sensitivity in accuracy rather than the proposed visual-context interaction. These limitations significantly weaken the broader contribution of the work.
[1] Fool Your (Vision and) Language Model with Embarrassingly Simple Permutations. ICML 2024.
[2] Benchmarking and Mitigating MCQA Selection Bias of Large Vision-Language Models. EMNLP 2025.
[3] Large Language Models Are Not Robust Multiple Choice Selectors. ICLR 2024.
[4] Option Symbol Matters: Investigating and Mitigating Multiple-Choice Option Symbol Bias of Large Language Models. NAACL 2025.
Questions for authors
None
2 comments
Gentle Reminder
Dear Reviewer NStU,
Thank you once again for your thoughtful review and the time you have devoted to our paper. As the discussion period is drawing to a close, we wanted to kindly follow up on our responses. We hope they have addressed your concerns. If anything remains unclear, we would be glad to provide further clarification promptly.
If you feel our responses have resolved your concerns, we would sincerely appreciate it if you could take them into account in your final evaluation.
Best regards,
The Authors
Response to Reviewer NStU (1/1)
Dear Reviewer NStU,
Thank you for recognizing our research question as relevant and our experimental design as careful. In the revised PDF, changes for your review are green and changes shared with other reviewers are purple, each tagged with your weakness W1 or W2.
W1. The novelty and depth are limited given [1]–[4], and the explanation remains behavioral.
A1: Thank you for this point. Our effect follows a route that the mechanisms in [1]–[4] cannot produce, and a new intervention locates part of it:
We added the intervention at the end of Section 5.2, with details and an attention-masking test in the new Appendix C.5, and related our work to [3, 4] in Section 2. Therefore, the scene reaches the answer partly through the label tokens on the option lines, a route distinct from the image-independent mechanisms in [1]–[4].
W2. The paper reads like an experimental report, and its findings are confined to a spelling task.
A2: Thank you for this comment. We reorganized the paper around one proposition, and we show a consequence beyond the spelling task:
We stated the proposition in Section 1, added the crossed test as the new Section 5.3, added the WorldCuisines results as a new paragraph and Table 2 in Section 7 with details in Appendices E.3 and E.4, and moved MMVP to Appendix E.2; Appendix F reports our first attempts on CLEVR and VQAv2. Therefore, the experiments form one testable account, and its consequence extends to an existing cultural benchmark.
Thank you again for your feedback, which has strengthened the manuscript. We hope our response addresses your concerns, and we kindly ask you to reconsider your evaluation.
Best,
Authors