- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper proposes Exact Visual Grounding (EVG), an answer-first pipeline for synthesizing perception-centric VQA data. For each task and difficulty level, it samples a target answer, renders a synthetic scene whose queried property exactly realizes that answer, and then fills in a question template. The result is annotation-free supervision across 12 Color, Counting, and Geometric Measurement tasks, plus matched missing-object negatives. Models are fully fine-tuned with a curriculum, and checkpoints are selected on a held-out Vero-600K panel. On Qwen3.5-9B, EVG raises a nine-benchmark macro-average from 60.06 to 63.90, and gains are also reported for Qwen3.5-4B, Gemma-4-12B, and the agentic DeepEyesV2-7B. Module grafting and layer-wise linear probing are used to argue that the changes span both the vision and language stacks and make visual properties more linearly accessible in middle-to-late LM layers.
Critical strengths
- Exact, cheap, and controllable supervision. Labels are fixed by the rendering parameters, so the pipeline produces noise-free supervision without annotators or upstream detectors. The calibrated answer distributions, randomized nuisance factors, matched missing-object negatives, and explicit validity checks show care in closing obvious shortcuts. The full recipe runs in about 10 hours on one GPU and ships with the generator, so it is easy to extend and reproduce.
- Targets a real and well-documented failure mode. The method directly addresses VLMs’ tendency to answer from priors rather than from pixels. The largest gains appear where that tendency is exposed: VLMs Are Biased (+10.1) and VLMs Are Blind (+5.4). The qualitative cases, such as the added star on the New Zealand flag and the 10-column Sudoku grid, show a real shift from memorized defaults to reading the image.
- Broad empirical coverage with useful controls. Results span two Qwen scales, a second backbone, and an agentic model, where the higher crop-tool rate is an interesting behavioral side effect. The paper also includes task-family, curriculum, and missing-ratio ablations, a same-recipe re-implementation of Rizzoli et al., and the explicit-visual-question subset analysis, which together go beyond headline numbers.
Critical weaknesses
- Novelty and the scope of “transfer” are overstated. Answer-first sampling is essentially standard procedural generation with a calibrated answer prior, and closely related work is missing. Euclid trains purely on synthetic geometric data with multi-stage curriculum training and reports generalization to novel shapes. VisOnlyQA releases 70k synthetic training instances and finds that fine-tuning gains are confined to certain tasks and models. In addition, the largest gains come from VLMs Are Blind and VLMs Are Biased, whose items (line intersections, grid rows/columns, counting stars or columns) closely mirror EVG’s Counting templates, so these results are better described as near-domain than as downstream transfer.
- Base models appear to be evaluated far from their operating point. The reported base scores for Qwen3.5-9B (50.78 MMMU-Val, 37.69 MMMU-Pro) and Qwen3.5-4B (47.11, 33.64) are roughly 28–33 points below the model card’s figures of 78.4/70.1 and 77.6/66.3 on MMMU/MMMU-Pro, and Qwen3.5 runs in thinking mode by default. Qwen3.5-4B also scores only 1.44% on VLMs Are Biased. EVG trains on short, format-constrained answers, so part of the gain may come from fixing formatting, response length, or answer parsing under this harness rather than from better perception. Whether the gains survive under the recommended inference settings is central to the claim and currently unknown.
- The mechanistic analyses do not support the causal story. The probes are trained on the same 12 tasks used for fine-tuning, at the last prompt token right before the answer. Higher late-layer decodability is therefore largely expected after supervising those answers, and it says little about why unrelated benchmarks improve. No control tasks are reported despite citing Hewitt & Liang, and a 20-iteration probe conflates accessibility with ease of optimization. The two analyses also conflict: grafting places the LM contribution mainly in layers 0–7 and almost none in layers 24–31 (86.89 vs. 86.50), while probing places the divergence after layer 16, and neither this tension nor the non-additive stack grafts (3.43 + 2.50 > 5.37) is addressed.
Questions for authors
N/A
6 comments
Authors' Rebuttal (2/4)
(A1 continued)
To further examine whether these precise-perception benefits complement broader vision-centric training, we conduct an additional experiment with Zebra-CoT. We train sequentially on Zebra-CoT and EVG, allocating half of the total training steps to each dataset while keeping the total training-step budget the same as the Zebra-CoT baseline. We denote this sequential training setup as Z+E.
Table 3: Results on the held-out EVG test set for Zebra-CoT and Z+E.
| Training data | D1 EM | D2 EM | D3 EM | Mean EM | Numeric MRE ↓ |
|---|---|---|---|---|---|
| Zebra-CoT | 36.17 | 23.75 | 13.75 | 24.56 | 34.02 |
| Z+E | 61.50 | 32.75 | 20.00 | 38.08 | 18.53 |
Table 4: Results on the nine VQA benchmarks for Zebra-CoT and Z+E.
| Training data | MMMU-Val | MMStar | AI2D | ChartQA | BLINK | HallusionBench | VLMs Are Biased | VLMs Are Blind | MMMU-Pro | Macro |
|---|---|---|---|---|---|---|---|---|---|---|
| Zebra-CoT | 58.78 | 70.59 | 88.12 | 86.00 | 68.34 | 54.51 | 22.27 | 81.87 | 44.28 | 63.86 |
| Z+E | 59.22 | 71.99 | 88.28 | 87.44 | 67.75 | 55.16 | 25.36 | 86.14 | 44.57 | 65.10 |
At the same total training-step budget, Z+E raises the nine-benchmark macro-average from 63.86% to 65.10% (+1.24 percentage points), improving eight of the nine benchmarks over Zebra-CoT alone. On the held-out EVG test set, mean exact-match accuracy increases from 24.56% to 38.08%, while numerical MRE decreases from 34.02% to 18.53%. Notably, VLMs Are Blind accuracy recovers from 81.87% to 86.14%, substantially mitigating the decline after Zebra-CoT training and bringing performance close to the Base score of 86.50%.
These results highlight EVG's distinct contribution to precise visual perception and show that this supervision can also complement Zebra-CoT training.
Regarding transfer, VLMs Are Blind and VLMs Are Biased evaluate fine-grained visual perception and reliance on visual evidence, which are central to EVG's training objectives. Improvements on these independently constructed benchmarks show that the targeted capabilities extend beyond our generated training tasks. Moreover, neither benchmark is limited to counting. VLMs Are Blind also tests overlap and contact, circled-letter recognition, path connectivity, and nested spatial structure. VLMs Are Biased additionally covers equality and attribute identification across optical illusions, flags, logos, animals, chess pieces, game boards, and patterned grids. Improvement also remains after excluding both benchmarks: EVG raises the macro-average over the other seven benchmarks from 62.74% to 65.47% (+2.74 pp). We will therefore state the main conclusion more precisely: EVG primarily strengthens exact low-level visual perception, and this targeted capability also benefits broader VQA tasks.
[1] Euclid: Supercharging Multimodal LLMs with Synthetic High-Fidelity Visual Descriptions. arXiv 2024.
[2] VisOnlyQA: Large Vision Language Models Still Struggle with Visual Perception of Geometric Information. COLM 2025.
[3] Unfolding Spatial Cognition: Evaluating Multimodal Models on Visual Simulations. ICLR 2026.
[4] Zebra-CoT: A Dataset for Interleaved Vision-Language Reasoning. ICLR 2026.
Authors' Rebuttal (3/4)
W2. Base models appear to be evaluated far from their operating point. The reported base scores for Qwen3.5-9B (50.78 MMMU-Val, 37.69 MMMU-Pro) and Qwen3.5-4B (47.11, 33.64) are roughly 28–33 points below the model card's figures of 78.4/70.1 and 77.6/66.3 on MMMU/MMMU-Pro, and Qwen3.5 runs in thinking mode by default. Qwen3.5-4B also scores only 1.44% on VLMs Are Biased. EVG trains on short, format-constrained answers, so part of the gain may come from fixing formatting, response length, or answer parsing under this harness rather than from better perception. Whether the gains survive under the recommended inference settings is central to the claim and currently unknown.
A2. We thank the reviewer for pointing out the discrepancy between our reported scores and the model card. Our original evaluations use the default configurations of the standard VLM evaluation platform LMMs-Eval, where the MMMU/MMMU-Pro tasks are evaluated in non-thinking mode, and the models are required to directly output the answers. Both Base and EVG achieve a 100% answer-parsing success rate on MMMU-Val and MMMU-Pro under these settings. Therefore, the accuracy gains cannot be explained by a reduction in answer-parsing failures. Although the complete evaluation settings behind the model-card scores were unavailable to us, we obtained comparable baseline results using thinking mode, greedy decoding, and a maximum output budget of 32,768 tokens. Below, we report Qwen3.5-9B results under these settings on MMMU/MMMU-Pro and BLINK-Twice, a reasoning benchmark focused on visual perception [1]:
| Reasoning Benchmark | Base | EVG |
|---|---|---|
| MMMU-Val | 78.33 | 78.33 |
| MMMU-Pro | 71.33 | 72.37 |
| BLINK-Twice | 41.49 | 43.60 |
As shown by the results, the gains are indeed smaller in thinking mode than in the non-thinking settings. However, EVG improves accuracy on MMMU-Pro and BLINK-Twice despite being trained exclusively on non-thinking SFT data. These results suggest that improvements in exact visual perception can also benefit reasoning.
Moreover, frontier performance on MMMU is approaching the reported expert reference: the 2026 AI Index reports 88.2% accuracy for the leading model, within 0.4 percentage points of the best human expert reference, implying limited space for further improvement [2]. Prior work [1] also shows that reasoning can support visual perception: such inference-time reasoning may partly compensate for perceptual weaknesses, helping explain the smaller additional gains from EVG in thinking mode.
Non-thinking inference also remains important for applications with tight latency and token budgets. Qwen3-VL explicitly supports both thinking and non-thinking variants to address different application requirements [3]. Furthermore, MME-CoT finds that chain-of-thought prompting can reduce accuracy on perception-heavy tasks through overthinking, reinforcing the value of evaluating direct-answer performance [4]. Therefore, the larger gains observed with EVG in non-thinking mode are practically relevant and directly support our central claim that exact visual supervision strengthens the extraction of fine-grained visual evidence. At the same time, the positive results in thinking mode show that this benefit is not limited to direct-answer inference.
In the revised manuscript, we will include these thinking-mode results and the corresponding analysis.
[1] BLINK-Twice: You see, but do you observe? A Reasoning Benchmark on Visual Perception. NeurIPS 2025.
[2] The 2026 AI Index Report. Stanford HAI 2026.
[3] Qwen3-VL Technical Report. arXiv 2025.
[4] MME-CoT: Benchmarking Chain-of-Thought in Large Multimodal Models for Reasoning Quality, Robustness, and Efficiency. ICML 2025.
Authors' Rebuttal (1/4)
We thank the reviewer for their valuable feedback.
W1. Novelty and the scope of "transfer" are overstated. Answer-first sampling is essentially standard procedural generation with a calibrated answer prior, and closely related work is missing. Euclid trains purely on synthetic geometric data with multi-stage curriculum training and reports generalization to novel shapes. VisOnlyQA releases 70k synthetic training instances and finds that fine-tuning gains are confined to certain tasks and models. In addition, the largest gains come from VLMs Are Blind and VLMs Are Biased, whose items (line intersections, grid rows/columns, counting stars or columns) closely mirror EVG's Counting templates, so these results are better described as near-domain than as downstream transfer.
A1. We thank the reviewer for raising concerns about the scope. We do not claim that answer-first sampling alone is a new procedural-generation primitive. Our contribution is the EVG framework, which combines exact target realization, calibrated answer distributions, 12 complementary perception tasks, three difficulty levels, matched missing-object cases, and curriculum training. We will clarify this scope and expand the related-work discussion to include Euclid [1], which studies synthetic geometric supervision and curriculum learning, and VisOnlyQA [2], which provides 70K synthetic examples for geometric perception.
We also add three equal-budget SFT controls using VisOnlyQA [2], STARE [3], and Zebra-CoT [4], with the same Qwen3.5-9B backbone and training recipe used for our method except for the difficulty curriculum. We evaluate all checkpoints on the held-out EVG test set and the same nine VQA benchmarks.
We additionally include a controlled Euclid [1] baseline on the nine VQA benchmarks, using the same backbone and optimization setup.
Table 1: Results on the held-out EVG test set. EM denotes exact-match accuracy, and MRE denotes mean relative error. All values are percentages.
Table 2: Results on the nine VQA benchmarks. Macro denotes the unweighted average across the nine benchmarks. All values are percentages.
Notably, Euclid exhibits a clear degradation in general-purpose visual-language performance after fine-tuning, consistent with the Euclid paper's observation that its geometry-focused training can induce out-of-domain forgetting. This limitation likely arises from Euclid's narrowly synthetic, geometry-centric data construction, which has limited semantic and visual diversity. In contrast, our EVG encompasses diverse colors, shapes, distractors and visual layouts, which significantly increases the visual diversity.
As shown in Tables 1 and 2, our method retains the highest macro-average across the nine VQA benchmarks and outperforms the VisOnlyQA, STARE, and Zebra-CoT controls by a large margin on the held-out EVG test set. These three controls achieve comparable general VQA performance. These datasets emphasize different capabilities: VisOnlyQA uses coarse labels for geometric perception, STARE focuses on spatial transformations and visual simulations, and Zebra-CoT covers broad visual reasoning over puzzles, diagrams, planning, and strategy games. EVG directly provides exact supervision for fine-grained visual properties across color, counting, and geometric measurement.
This distinction is reflected in precise-perception performance. Relative to Base, fine-tuning on VisOnlyQA, STARE, and Zebra-CoT changes mean exact-match accuracy on the held-out EVG test set by -2.60, -3.16, and -5.57 percentage points, respectively, whereas our method improves it by 17.73 percentage points. Fine-tuning on the three alternative datasets also reduces accuracy on VLMs Are Blind from the Base score of 86.50% to 81.87%-83.35%, whereas our method increases it to 91.87%. Thus, improvements in broad VQA performance do not by themselves reproduce the precise-perception gains obtained with our method.