- Refusal is a tunable policy, not a lack of grounding, and the paper never tests the baseline that would tune it.
- Escalating is a sensible default. Flagging a conflict instead of guessing is how models are trained to avoid hallucinated or incorrect answers. When text and image disagree, the agent cannot know which is wrong, and a wrong silent correction acts on a bad value and removes the alarm.
- Which source wins is a deployment choice, not a measure of quality. The paper hard-codes "the image is right". In many settings the text should win: an instruction carrying a newer value makes a chart or screenshot stale. Treating (C) as "safe but useless"and crediting only (B) scores models against one arbitrary policy.
- The missing baseline is a policy prompt. If escalation is a policy, default internalized by the model, the first test is to state the policy: give the agent a role in which it is authorized to override the text using the image, and measure VisAcc. The paper skips this. It goes straight to a four-step CoVe scaffold. Without that baseline, there is no evidence that the four steps are needed, or that CoVe "recovers a latent ability" (Sec. 5.1) rather than simply telling the model what policy it should be following.
- The reported metrics do not support "CoVe improves every model". Blind-follow under CoVe is never reported. The reader can only infer it from SAFE = 1 − BFR : Gemini's SAFE falls from 1.00 to 0.54 (Sec. 5.3), which implies BFR ≈ 0.46, about as large as its VisAcc of 0.49.
- SAFE also counts wrong overrides (FakeOverride) as safe. A VisAcc gain alone therefore cannot establish an improvement; BFR and FakeOverride under each condition are needed
-
The dataset is stale and narrow: one image type, one question type.
-
Old sources. TextVQA [1] (2019) is largely OCR-solvable and has ambiguous answers; harder text-rich benchmarks such as ConTextual [3] were built to address this. ChartQA [2] overstates chart understanding, as CharXiv [4] shows.
-
Narrow image scenarios. Only scene-text photos and charts are used. There are no documents (DocVQA [5]), screens/GUIs (ScreenSpot [6]), natural scenes without text, or real-world noisy captures (VizWiz [7]; MME-RealWorld [8]), although the motivation is agents reading "a screen, a gauge, or a chart" (Sec. 1).
-
Narrow question types. Every conflict is a single extracted string. There are no conflicts over:
- counting (TallyQA [9]);
- spatial relations (What'sUp [10]);
- colour, attributes and fine detail (V* Bench [11]);
- low-level perception (MMVP [12]; CV-Bench [13]; BLINK [14]);
- GUI state [6].
The "difficulty stratification" (Table 6) stratifies by answer type (word/number/alphanumeric), not question type, and the cross-distribution claim (Sec. 4.2) varies only the image source.
-
Small scale despite trivial generation. Items are a template plus a sampled wrong value, yet the paper uses only 750 pairs. The gold audit is 70 items checked by Gemini-2.5-Pro, itself an evaluated model.
-
The writing and reporting make the paper hard to follow and hard to trust.
- Key figures are missing. Fig. 1 is incomplete / illegible and Fig2 has placeholders.
- Too many metrics defined that obscure the reader from understanding how VisAcc works, their key proposal. Interplay between the metrics should also be mentioned up front. For instance VoR = VisAcc + FakeOverride, because at place The above relation is not followed. (Sec. 4.3 (0.16/0.16/0.14) ≠ VOR − VisAcc from Table 2 (0.02/0.05/0.05).
- Inconsistencies in results, like Table 5 and Table 4 report different VisAcc for the same CoVe condition (Gemini 0.86 vs. 0.49). ?
References
[1] Singh et al., "Towards VQA Models That Can Read," CVPR 2019.
[2] Masry et al., "ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning," Findings of ACL 2022.
[3] Wadhawan et al., "ConTextual: Evaluating Context-Sensitive Text-Rich Visual Reasoning in Large Multimodal Models," ICML 2024.
[4] Wang et al., "CharXiv: Charting Gaps in Realistic Chart Understanding in Multimodal LLMs," NeurIPS 2024 (Datasets & Benchmarks).
[5] Mathew et al., "DocVQA: A Dataset for VQA on Document Images," WACV 2021.
[6] Cheng et al., "SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents," ACL 2024 (ScreenSpot).
[7] Gurari et al., "VizWiz Grand Challenge: Answering Visual Questions from Blind People," CVPR 2018.
[8] Zhang et al., "MME-RealWorld: Could Your Multimodal LLM Challenge High-Resolution Real-World Scenarios that are Difficult for Humans?," ICLR 2025.
[9] Acharya et al., "TallyQA: Answering Complex Counting Questions," AAAI 2019.
[10] Kamath et al., "What's 'up' with Vision-Language Models? Investigating Their Struggle with Spatial Reasoning," EMNLP 2023.
[11] Wu & Xie, "V*: Guided Visual Search as a Core Mechanism in Multimodal LLMs," CVPR 2024.
[12] Tong et al., "Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs," CVPR 2024.
[13] Tong et al., "Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs," NeurIPS 2024 (CV-Bench).
[14] Fu et al., "BLINK: Multimodal Large Language Models Can See but Not Perceive," ECCV 2024.