- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper studies how VLMs estimate temporal bands from images and how their answers are influenced by visual evidence or semantic knowledge. The authors devise a behavioral task that asks VLMs about an image depicting an action and examines how the prompt influences their forced-choice estimates of the approximate temporal band. They find that VLMs’ temporal-band estimates follow language-consistent answers.
Critical strengths
None identified.
Critical weaknesses
It is unclear why temporal reasoning, specifically temporal-band estimation, is an interesting question to study. I do not understand which aspects of VLMs this investigation helps us understand. The paper devotes substantial space to reporting paired response transitions between five-way and six-way choices, but the motivation for this comparison and its takeaway remain unclear.
I find the paper difficult to comprehend. The motivation, problem setup, and methods are poorly explained. The paper reports numerous evaluations, but what these evaluations measure and why particular design choices were made are very unclear. The choices of prompt format and the five-way condition appear arbitrary, and the central comparison between five-way and six-way forced-choice prompts is poorly motivated.
Questions for authors
What do the authors mean by “temporal band” and “temporal-scale estimation”? These terms need clear definitions; their current meanings are vague.
Why is temporal-band estimation an interesting question to study, and what does it help us understand about VLMs?
What motivates the prompt design, the five-way condition, and the comparison between five-way and six-way forced-choice prompts?
Why are paired response transitions central to the analysis, and what should readers take away from them?
How is the text-only baseline evaluated on VLMs?
Regarding multimodal interactions, please refer to the visual saliency paper by Tekker et al.: https://arxiv.org/pdf/2610.12650.
We thank the reviewer for the careful reading. We provide clarifications and new analyses addressing each concern below.
Scientific question and methodological stakes (W1, Q2). Forced-choice evaluation under image-text conflict is the dominant paradigm for measuring VLM modality preference in the current literature. The standard interpretation of a language-consistent forced-choice answer is that the model disregarded the visual evidence. Our paper tests this interpretation by extending the response space to include an explicit mismatch option. On the same items, most answers classified as language-consistent under forced choice become explicit mismatch reports when this option is available. The paper identifies a methodological problem in the dominant evaluation setup and provides behavioral and mechanistic evidence for reinterpreting its results.
Definitions (Q1). Temporal-band estimation is the task of estimating how long a visible process has been ongoing to reach its current state, with answer choices corresponding to five ordinal duration bands: A (seconds-minutes), B (hours-days), C (weeks-months), D (years-decades), E (decades-millennia). We chose this task because natural processes have characteristic durations. Water boiling is a minutes-scale process, while mountain erosion is a millennia-scale process. When the image depicts one process, and the question specifies a different one, visual evidence and process-description priors support different temporal bands. The ordinal structure also permits measuring conflict magnitude by band-distance.
Prompt design (W2, Q3). The two prompt formats isolate the role of the response space. Both use identical image-question pairs and the same five temporal bands (A–E). The six-way format adds option F, “The described process is not visible in the image.” This is the minimal change needed to test whether a language-consistent response under forced choice reflects modality preference or the absence of a mismatch option. The five-way condition reproduces the forced-choice setup of earlier modality-preference evaluations, while the six-way condition allows the model to report a mismatch explicitly.
Paired response transitions (Q4). Aggregate rates do not identify which individual predictions change between formats. We conducted a new paired item-level transition analysis on the same 500 contradictory examples per model.
L/V denote language-/vision-consistent responses; F denotes mismatch. Rates are conditional on the five-way response.
Across models, 55.4 to 90.2 percent of language-consistent five-way responses and 45.3 to 85.3 percent of vision-consistent responses switch to F when the option becomes available. A language-consistent forced-choice answer, interpreted in earlier studies as disregard for visual evidence, usually becomes an explicit mismatch report when the response space permits it. However, 8.3 to 42.9 percent of language-consistent responses persist under the six-way format, indicating that the forced-choice setup obscures mismatch detection without fully accounting for language-consistent responding.
Text-only baseline (Q5). The text-only baseline, specified in Appendix A.2.5 of the paper, uses the standard five-way prompt with the image removed. The process description, answer choices, and output instruction remain unchanged. Removing the image reduces standard-condition accuracy by 0 to 18 percentage points across the five models. Process descriptions alone support many correct temporal-band answers, which bears on how standard-condition accuracy should be interpreted.
Relation to Teker et al. (Q6). Teker et al. (2026) study word-color interference through the Stroop paradigm. They evaluate contrastive and generative VLMs, but their representational-similarity and concept-steering analyses focus on CLIP. Our work differs in four respects. First, we probe and patch hidden states in five generative VLMs from four language-model families. Second, query-token activation patching tests causal influence on answer selection, whereas their interventions modify CLIP image embeddings and measure similarity changes. Third, we analyze ordinal temporal-band distance; their study varies word-color congruence and visual salience. Fourth, we compare matched prompts with and without an explicit mismatch option. Although Teker et al. categorize free-form responses mentioning both cues, they do not manipulate the availability of a dedicated mismatch response. We will cite their study in Related Work.
Writing clarity. We will revise the introduction and methods to state the research question earlier, define temporal bands, and explain the five-way/six-way comparison.