- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies where and why grounded reasoning (outputting text coordinates during chain-of-thought) helps MLLMs. It introduces a decode-time coordinate randomization test to measure whether models actually use their grounding, and finds that many open grounded models do not. Through controlled training-recipe ablations, it shows that grounded reasoning outperforms text-only reasoning on counting but not on spatial reasoning or perception. Using synthetic tasks, it argues that grounding helps mainly when a task requires undescribable references, and that most existing benchmarks do not. Based on this finding, it proposes BrickBench, a benchmark that requires undescribable references, on which frontier models score below 20%.
Critical strengths
- The motivation of this paper is clear, interesting, and important
- The finding that grounded reasoning outperforms text-only reasoning in counting and undescriable reference is interesting and valuable, and it is well-supported.
- BrickBench is novel and could be important for the community. Most existing visual reasoning benchmarks can be solved with describable text references, so they cannot show whether grounded reasoning helps
Critical weaknesses
W1. Would the decode-time randomization test and the undescribable-reference findings also hold for other forms of grounding, such as tool-based crop/zoom or visual-token references (DeepEyes, v1, ClawMachine) [1-3]? For example, could the probe tasks in Sec. 4 be run with a crop/zoom or visual-token variant alongside text and coordinate CoT?
W2. BrickBench’s low frontier scores may not reflect a lack of grounded reasoning. In Table 4, text-only fine-tuning raises Qwen3.5-9B from 5.2 to 61.2, far above all frontier models (14.4 - 18.9). Qwen3.5 itself, which is explicitly trained for grounded reasoning (Sec 3.2), scores only 5.2 zero-shot. This suggests that frontier failures may stem largely from out-of-distribution data or the precise point-based answer format, rather than from the absence of grounded reasoning. Grounded fine-tuning does outperform text fine-tuning (82.9 vs. 61.2), but showing that the benchmark specifically measures undescribable-reference ability would require separating format effects from reference effects.
[1] Zheng et al., 2026, DeepEyes: Incentivizing “Thinking with Images” via Reinforcement Learning. ICLR 2026. https://arxiv.org/abs/2505.14362
[2] Chung et al., 2025 Don’t Look Only Once: Towards Multimodal Interactive Reasoning with Selective Visual Revisitation. arXiv 2025. https://arxiv.org/abs/2505.18842
[3] Ma et al., 2025, ClawMachine: Learning to Fetch Visual Tokens for Referential Comprehension. ICLR 2025. https://arxiv.org/abs/2406.11327
Questions for authors
-
Sec 3.1 argues that post-hoc coordinate randomization is insufficient because later text may already encode information derived from the coordinates, and introduces decode-time randomization instead. I agree with the reasoning, but it is not shown empirically. Could the authors compare post-hoc and decode-time randomization on the same checkpoints (e.g., Table 1) to show how much the two tests differ?
-
Could the authors compare the attention dynamics of the grounded and text-only checkpoints in Table 3? For example, do grounded checkpoints attend more to the image regions referenced by their coordinates, and does this differ between counting and spatial reasoning tasks? This would help explain why grounding helps counting but not spatial reasoning.
4 comments
W1 followup Would the decode-time randomization test and the undescribable-reference findings also hold for other forms of grounding, such as tool-based crop/zoom or visual-token references (DeepEyes, v1, ClawMachine) [1-3]? For example, could the probe tasks in Sec. 4 be run with a crop/zoom or visual-token variant alongside text and coordinate CoT?
Please see the reply to pqhd for our decode randomization findings for other forms of grounding. Note that ClawMachine does not have released weights, so we run our analysis on v1, PixelReasoner, and DeepEyes. We also rerun the probe tasks with crop/zoom variants alongside text and coordinate CoT. We find that our findings replicate for tool-based crop/zoom methods. When labels are not provided, crop/zoom grounded reasoning outperforms text CoT, but the advantage disappears or reverses when labels are provided. Crop/zoom performs better than crop CoT on the wire-tracing task, likely because the model is better able to analyze small areas where wires cross each other. Otherwise, crop/zoom CoT performs similarly to coordinate CoT.
| Probe | Labels | Direct answer | Text CoT | Coordinate CoT | Crop/zoom CoT |
|---|---|---|---|---|---|
| Counting | No | 34.4 | 1.8 | 95.2 | 92.4 |
| Yes | 31.2 | 98.0 | 97.2 | 98.6 | |
| Wire tracing | No | 12.0 | 24.4 | 69.3 | 83.2 |
| Yes | 10.8 | 91.4 | 79.4 | 83.2 | |
| Arrow following | No | 9.4 | 62.4 | 75.4 | 77.6 |
| Yes | 9.4 | 98.4 | 71.4 | 76.0 |
Q1 followup Could the authors compare post-hoc and decode-time randomization on the same checkpoints (e.g., Table 1) to show how much the two tests differ?
We compare decode-time randomization to post-hoc randomization. As expected, post-hoc randomization generally does not yield significant effect on performance. Notably, while the decode-time randomization effect yields significant decreases in performance on counting on our reflection models and TreeVGR-RL, post-hoc randomization shows no performance drop, as predicted.
| Model | MHS decode-time | MHS post-hoc | SMQA decode-time | SMQA post-hoc | V* decode-time | V* post-hoc | CountQA decode-time | CountQA post-hoc |
|---|---|---|---|---|---|---|---|---|
| TreeVGR-7B-CI | −0.35 ± 0.50 | +0.04 ± 0.40 | −0.61 ± 0.79 | −0.41 ± 0.40 | −2.09 ± 1.30 | +0.42 ± 1.30 | +0.08 ± 0.74 | −0.10 ± 0.46 |
| TreeVGR-7B-RL | +0.52 ± 0.35 | −0.45 ± 0.29 | +0.11 ± 0.36 | −0.45 ± 0.45 | −0.94 ± 0.98 | −0.31 ± 0.59 | −2.66 ± 0.51 | +0.26 ± 0.64 |
| iVGR-8B | −0.95 ± 0.39 | −0.68 ± 0.25 | −0.71 ± 0.48 | −0.22 ± 0.56 | −0.73 ± 0.99 | −0.73 ± 0.55 | −2.78 ± 0.69 | −0.34 ± 0.78 |
| DeepEyes-7B | +0.45 ± 0.74 | −0.21 ± 0.78 | +0.06 ± 0.51 | +0.28 ± 0.69 | −0.63 ± 0.77 | +1.88 ± 1.31 | −0.80 ± 0.51 | +0.12 ± 0.51 |
| Appended (ours) | −2.09 ± 0.33 | −0.12 ± 0.24 | −2.04 ± 0.77 | +0.02 ± 0.66 | −1.78 ± 1.54 | +0.00 ± 1.30 | −0.12 ± 0.68 | −0.02 ± 0.65 |
| Reflection (ours) | −3.60 ± 0.59 | −0.37 ± 0.74 | −2.40 ± 0.82 | −0.07 ± 0.67 | −3.04 ± 1.01 | +0.00 ± 0.80 | −4.26 ± 0.91 | −0.12 ± 0.85 |
Q2 followup Could the authors compare the attention dynamics of the grounded and text-only checkpoints in Table 3? For example, do grounded checkpoints attend more to the image regions referenced by their coordinates, and does this differ between counting and spatial reasoning tasks?
We find that when models are outputting coordinates, they attend to the mentioned regions. We perform an experiment where we take tokens near generated coordinates and compare the attention on the location of the coordinates, compared to attention at random locations. As seen below, the attention is concentrated on locations near the outputted coordinate, which is expected.
| Benchmark | Attention near the coordinate being written (× compared to random locations) |
|---|---|
| CountQA | 4.7 |
| SpatialMQA | 4.8 |
When analyzing attention on the image across the entire sequence, we find that grounding actually shifts attention away from the image on counting tasks, likely because producing the final answer requires performing a sum operation over the number of text coordinates outputted. More detailed attention analysis would be a promising avenue for future work, as there are likely interactions from attending to both text coordinate tokens and image tokens together.
| Benchmark | Attention on the image, text-only | Attention on the image, grounded |
|---|---|---|
| CountQA | 17.6% | 16.7% |
| SpatialMQA | 7.3% | 8.0% |
Dear Reviewer 72BK,
Thank you again for your review. As the discussion period nears its end, we wanted to check whether our response addresses your concerns.
Thank you for the response and the additional experiments. Most of my concerns are resolved. However, the concern regarding BrickBench (W2) is not resolved. Text-only fine-tuning raises Qwen3.5-9B from 5.2 to 61.2, far above all frontier models, without any grounding. This suggests that the low frontier scores mainly reflect unfamiliarity with the task rather than the inability to track undescribable references, so it remains unclear what BrickBench actually measures. Also, the paper introduces BrickBench to close the gap that existing benchmarks cannot demonstrate the benefit of grounded reasoning (L87-90, L432-436), but the response states that it is not meant to measure grounded reasoning specifically.
Hence, I will keep my score.
As mentioned in our reply to pqhd, Table 1 of our paper applies the decode-time randomization test (modified to edit both the text and the displayed crop) to DeepEyes, and found a similar lack of grounding usage on spatial reasoning, counting, and perception. We will perform more extensive evaluation across other grounded reasoning variants and report our results in the discussion period.
We appreciate the reviewer’s concern on this. However, we would like to clarify that BrickBench's goal is not to measure specifically grounded reasoning, but instead to measure the more general ability to precisely track un-describable references. Because BrickBench tracks this higher-level capability, it is a flexible benchmark that can be used to study improvements in other reasoning techniques. Grounded reasoning is one valid strategy, but BrickBench could measure the benefit of other techniques such as latent reasoning, or looped models. We will edit the paper to make this point more clear. See our discussion with pqhd for further discussion on whether BrickBench images are out of distribution.
If the primary concern is the point-based answer format, we also note that the “Fine-grained counting” subsplit of the benchmark is not point-based, and takes a text integer value as its answer format. As seen in Table 4 of our paper (reproduced below), fine-grained counting performance is similar or lower than other task splits that require point-based outputs, suggesting that the primary difficulty is not in the point-based answer format.
We will perform these tests and report our results in the discussion period.
We will perform these analyses of attention dynamics and report our results in the discussion period.
We would deeply appreciate it if you could clarify whether these clarifications and proposed experiments would address your concerns.