acceptodds
Under review as a conference paper at ICLR 2027

Where and Why is Grounded Reasoning Useful?

Abstract

Grounded reasoning, where a model outputs text coordinates of relevant locations in chain-of-thought reasoning, is a common strategy for solving complex visual reasoning problems. However, the benefits and limitations of grounded reasoning remain understudied. We study where and why grounded reasoning is most useful. To do so, we introduce a chain-of-thought analysis to measure how much a model uses its outputted grounding in its thinking trace and perform a detailed ablation study of a grounded reasoning training recipe. We find that grounded reasoning improves counting but shows no significant gain over baselines without coordinate grounding across a wide range of spatial reasoning and perception benchmarks. Using several synthetic tasks, we show that grounded reasoning is most useful when a task requires undescribable references to visual concepts that are difficult to describe in language. However, we observe that many existing visual reasoning benchmarks are solvable with only describable text references. To close this gap, we introduce BrickBench, a new benchmark that requires undescribable visual references to solve.

Primary area:
foundation or frontier models, including LLMs
Keywords:
grounded reasoning, vision language models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_72BKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies where and why grounded reasoning (outputting text coordinates during chain-of-thought) helps MLLMs. It introduces a decode-time coordinate randomization test to measure whether models actually use their grounding, and finds that many open grounded models do not. Through controlled training-recipe ablations, it shows that grounded reasoning outperforms text-only reasoning on counting but not on spatial reasoning or perception. Using synthetic tasks, it argues that grounding helps mainly when a task requires undescribable references, and that most existing benchmarks do not. Based on this finding, it proposes BrickBench, a benchmark that requires undescribable references, on which frontier models score below 20%.

Critical strengths
  1. The motivation of this paper is clear, interesting, and important
  2. The finding that grounded reasoning outperforms text-only reasoning in counting and undescriable reference is interesting and valuable, and it is well-supported.
  3. BrickBench is novel and could be important for the community. Most existing visual reasoning benchmarks can be solved with describable text references, so they cannot show whether grounded reasoning helps
Critical weaknesses

W1. Would the decode-time randomization test and the undescribable-reference findings also hold for other forms of grounding, such as tool-based crop/zoom or visual-token references (DeepEyes, v1, ClawMachine) [1-3]? For example, could the probe tasks in Sec. 4 be run with a crop/zoom or visual-token variant alongside text and coordinate CoT?

W2. BrickBench’s low frontier scores may not reflect a lack of grounded reasoning. In Table 4, text-only fine-tuning raises Qwen3.5-9B from 5.2 to 61.2, far above all frontier models (14.4 - 18.9). Qwen3.5 itself, which is explicitly trained for grounded reasoning (Sec 3.2), scores only 5.2 zero-shot. This suggests that frontier failures may stem largely from out-of-distribution data or the precise point-based answer format, rather than from the absence of grounded reasoning. Grounded fine-tuning does outperform text fine-tuning (82.9 vs. 61.2), but showing that the benchmark specifically measures undescribable-reference ability would require separating format effects from reference effects.

[1] Zheng et al., 2026, DeepEyes: Incentivizing “Thinking with Images” via Reinforcement Learning. ICLR 2026. https://arxiv.org/abs/2505.14362

[2] Chung et al., 2025 Don’t Look Only Once: Towards Multimodal Interactive Reasoning with Selective Visual Revisitation. arXiv 2025. https://arxiv.org/abs/2505.18842

[3] Ma et al., 2025, ClawMachine: Learning to Fetch Visual Tokens for Referential Comprehension. ICLR 2025. https://arxiv.org/abs/2406.11327

Questions for authors
  1. Sec 3.1 argues that post-hoc coordinate randomization is insufficient because later text may already encode information derived from the coordinates, and introduces decode-time randomization instead. I agree with the reasoning, but it is not shown empirically. Could the authors compare post-hoc and decode-time randomization on the same checkpoints (e.g., Table 1) to show how much the two tests differ?

  2. Could the authors compare the attention dynamics of the grounded and text-only checkpoints in Table 3? For example, do grounded checkpoints attend more to the image regions referenced by their coordinates, and does this differ between counting and spatial reasoning tasks? This would help explain why grounding helps counting but not spatial reasoning.

4 comments

W1. Would the decode-time randomization test and the undescribable-reference findings also hold for other forms of grounding, such as tool-based crop/zoom or visual-token references (DeepEyes, v1, ClawMachine) [1-3]? For example, could the probe tasks in Sec. 4 be run with a crop/zoom or visual-token variant alongside text and coordinate CoT?

As mentioned in our reply to pqhd, Table 1 of our paper applies the decode-time randomization test (modified to edit both the text and the displayed crop) to DeepEyes, and found a similar lack of grounding usage on spatial reasoning, counting, and perception. We will perform more extensive evaluation across other grounded reasoning variants and report our results in the discussion period.

W2 BrickBench’s low frontier scores may not reflect a lack of grounded reasoning. Fontier failures may stem largely from out-of-distribution data or the precise point-based answer format, rather than from the absence of grounded reasoning.

We appreciate the reviewer’s concern on this. However, we would like to clarify that BrickBench's goal is not to measure specifically grounded reasoning, but instead to measure the more general ability to precisely track un-describable references. Because BrickBench tracks this higher-level capability, it is a flexible benchmark that can be used to study improvements in other reasoning techniques. Grounded reasoning is one valid strategy, but BrickBench could measure the benefit of other techniques such as latent reasoning, or looped models. We will edit the paper to make this point more clear. See our discussion with pqhd for further discussion on whether BrickBench images are out of distribution.

If the primary concern is the point-based answer format, we also note that the “Fine-grained counting” subsplit of the benchmark is not point-based, and takes a text integer value as its answer format. As seen in Table 4 of our paper (reproduced below), fine-grained counting performance is similar or lower than other task splits that require point-based outputs, suggesting that the primary difficulty is not in the point-based answer format.

ModelCrop IDFine-grained countingDuplicateUnpairedOverall
GPT-5.6 Sol35.28.311.620.418.9
Gemini 3.8 Flash20.415.58.817.615.6
Claude Opus 526.810.78.411.614.4
Qwen 3.8 Max6.42.013.212.48.5

Q1 Could the authors compare post-hoc and decode-time randomization on the same checkpoints (e.g., Table 1) to show how much the two tests differ?

We will perform these tests and report our results in the discussion period.

Q2 Could the authors compare the attention dynamics of the grounded and text-only checkpoints in Table 3? For example, do grounded checkpoints attend more to the image regions referenced by their coordinates, and does this differ between counting and spatial reasoning tasks?

We will perform these analyses of attention dynamics and report our results in the discussion period.

We would deeply appreciate it if you could clarify whether these clarifications and proposed experiments would address your concerns.

W1 followup Would the decode-time randomization test and the undescribable-reference findings also hold for other forms of grounding, such as tool-based crop/zoom or visual-token references (DeepEyes, v1, ClawMachine) [1-3]? For example, could the probe tasks in Sec. 4 be run with a crop/zoom or visual-token variant alongside text and coordinate CoT?

Please see the reply to pqhd for our decode randomization findings for other forms of grounding. Note that ClawMachine does not have released weights, so we run our analysis on v1, PixelReasoner, and DeepEyes. We also rerun the probe tasks with crop/zoom variants alongside text and coordinate CoT. We find that our findings replicate for tool-based crop/zoom methods. When labels are not provided, crop/zoom grounded reasoning outperforms text CoT, but the advantage disappears or reverses when labels are provided. Crop/zoom performs better than crop CoT on the wire-tracing task, likely because the model is better able to analyze small areas where wires cross each other. Otherwise, crop/zoom CoT performs similarly to coordinate CoT.

ProbeLabelsDirect answerText CoTCoordinate CoTCrop/zoom CoT
CountingNo34.41.895.292.4
Yes31.298.097.298.6
Wire tracingNo12.024.469.383.2
Yes10.891.479.483.2
Arrow followingNo9.462.475.477.6
Yes9.498.471.476.0

Q1 followup Could the authors compare post-hoc and decode-time randomization on the same checkpoints (e.g., Table 1) to show how much the two tests differ?

We compare decode-time randomization to post-hoc randomization. As expected, post-hoc randomization generally does not yield significant effect on performance. Notably, while the decode-time randomization effect yields significant decreases in performance on counting on our reflection models and TreeVGR-RL, post-hoc randomization shows no performance drop, as predicted.

ModelMHS decode-timeMHS post-hocSMQA decode-timeSMQA post-hocV* decode-timeV* post-hocCountQA decode-timeCountQA post-hoc
TreeVGR-7B-CI−0.35 ± 0.50+0.04 ± 0.40−0.61 ± 0.79−0.41 ± 0.40−2.09 ± 1.30+0.42 ± 1.30+0.08 ± 0.74−0.10 ± 0.46
TreeVGR-7B-RL+0.52 ± 0.35−0.45 ± 0.29+0.11 ± 0.36−0.45 ± 0.45−0.94 ± 0.98−0.31 ± 0.59−2.66 ± 0.51+0.26 ± 0.64
iVGR-8B−0.95 ± 0.39−0.68 ± 0.25−0.71 ± 0.48−0.22 ± 0.56−0.73 ± 0.99−0.73 ± 0.55−2.78 ± 0.69−0.34 ± 0.78
DeepEyes-7B+0.45 ± 0.74−0.21 ± 0.78+0.06 ± 0.51+0.28 ± 0.69−0.63 ± 0.77+1.88 ± 1.31−0.80 ± 0.51+0.12 ± 0.51
Appended (ours)−2.09 ± 0.33−0.12 ± 0.24−2.04 ± 0.77+0.02 ± 0.66−1.78 ± 1.54+0.00 ± 1.30−0.12 ± 0.68−0.02 ± 0.65
Reflection (ours)−3.60 ± 0.59−0.37 ± 0.74−2.40 ± 0.82−0.07 ± 0.67−3.04 ± 1.01+0.00 ± 0.80−4.26 ± 0.91−0.12 ± 0.85

Q2 followup Could the authors compare the attention dynamics of the grounded and text-only checkpoints in Table 3? For example, do grounded checkpoints attend more to the image regions referenced by their coordinates, and does this differ between counting and spatial reasoning tasks?

We find that when models are outputting coordinates, they attend to the mentioned regions. We perform an experiment where we take tokens near generated coordinates and compare the attention on the location of the coordinates, compared to attention at random locations. As seen below, the attention is concentrated on locations near the outputted coordinate, which is expected.

BenchmarkAttention near the coordinate being written (× compared to random locations)
CountQA4.7
SpatialMQA4.8

When analyzing attention on the image across the entire sequence, we find that grounding actually shifts attention away from the image on counting tasks, likely because producing the final answer requires performing a sum operation over the number of text coordinates outputted. More detailed attention analysis would be a promising avenue for future work, as there are likely interactions from attending to both text coordinate tokens and image tokens together.

BenchmarkAttention on the image, text-onlyAttention on the image, grounded
CountQA17.6%16.7%
SpatialMQA7.3%8.0%

Dear Reviewer 72BK,

Thank you again for your review. As the discussion period nears its end, we wanted to check whether our response addresses your concerns.

Thank you for the response and the additional experiments. Most of my concerns are resolved. However, the concern regarding BrickBench (W2) is not resolved. Text-only fine-tuning raises Qwen3.5-9B from 5.2 to 61.2, far above all frontier models, without any grounding. This suggests that the low frontier scores mainly reflect unfamiliarity with the task rather than the inability to track undescribable references, so it remains unclear what BrickBench actually measures. Also, the paper introduces BrickBench to close the gap that existing benchmarks cannot demonstrate the benefit of grounded reasoning (L87-90, L432-436), but the response states that it is not meant to measure grounded reasoning specifically.

Hence, I will keep my score.

Reviewer_pqhdOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper systematically investigates where and why textual-coordinate grounded reasoning is useful for VLMs. It introduces decode-time grounding randomization with KV-cache reset as a cleaner faithfulness test, finding that open grounded models (TreeVGR, iVGR, DeepEyes) barely rely on grounding except on counting tasks. Through controlled training ablations on Qwen3-VL-8B, the authors show that even high-quality grounding data with high usage rates yields significant gains over strong text-only baselines only on counting. Using synthetic probe tasks with a with/without label design, they isolate "undescribable visual references" as the locus of grounding's value, and build BrickBench where frontier models score below 20% while fine-tuned grounded models reach 82.9%.

Critical strengths
  1. This paper directly questions the prevailing "grounding always helps" narrative and offers a more complete causal explanation than iVGR's.
  2. Decode-time randomization and KV-cache reset fixes a genuine flaw in post-hoc CoT editing tests, yielding a clean, reusable faithfulness probe.
  3. The with and without text-label controlled probes cleanly isolate reference describability as the key variable, and the performance reversal is compelling evidence, and multi-seed clustered-CI reporting is above the field's norm.
Critical weaknesses
  1. All self-training experiments rely on a single backbone (Qwen3-VL-8B) with only 6,368 SFT samples, so the core negative conclusion cannot rule out insufficient data scale or a suboptimal training recipe as the underlying cause.
  2. Concerns remain about BrickBench's validity. Its programmatically generated, Nano-Banana-restyled brick-stack tasks are highly artificial and far removed from real-world visual reasoning. Moreover, the sharp gap—frontier models below 20% versus 82.9% for the fine-tuned model trained on pre-style-transfer images—offers weak evidence for the benchmark's difficulty, given that the latter is essentially an in-domain evaluation.
  3. The term "grounded reasoning" in this paper refers only to textual-coordinate grounding; the conclusions are overgeneralized, as no controlled comparison is made against other grounding variants such as crop/zoom or feature replay.
Questions for authors

See the weaknesses.

4 comments

W1 All self-training experiments rely on a single backbone (Qwen3-VL-8B) with only 6,368 SFT samples.

We will perform the self-training experiments across other VLM backbones and report our results in the discussion period. On dataset size, we note that significantly larger grounded reasoning models, such as in Thinking with Visual Primitives (284B total parameters and 13B active parameters) [1], only uses 9,000 high quality SFT samples in their spatial reasoning subset (of which a significant portion is synthetic CLEVR images). This reference from Thinking with Visual Primitives, which also studies text-coordinate grounded reasoning, suggests the scale of our dataset should be sufficient. We are happy to further discuss this point, if there is disagreement.

W2 Concerns remain about BrickBench's validity. Its programmatically generated, Nano-Banana-restyled brick-stack tasks are highly artificial and far removed from real-world visual reasoning.

This is a reasonable concern. However, BrickBench provides useful measures of model capability, even if it is synthetic. We believe that tasks in BrickBench are simple for humans to solve while being difficult for models, and that BrickBench provides significant value in measuring this gap. We will perform a human study and report results in the discussion period to demonstrate this.

Moreover, many VLM benchmarks, such as BabyVision [2], use synthetic, non-natural images to test visual reasoning, and have been adopted as standard evaluations for new model releases (Qwen3.8-Max, Inkling, and Muse). These models are able to score very high on BabyVision but poorly on BrickBench (76.9% vs 18.9% for GPT 5.6 Sol), suggesting that the difficulty is the task, not the image style.

If the concern that the Nano-Bannana image distribution is particularly out of distribution, we also provide re-evaluations of the strongest models on the original 2D render. Even for the 2D graphics renders, which are simpler and cleaner, the models still struggle. The difficulty across both image styles suggests that the fundamental difficulty comes from the task instead of image style.

ModelRaw synthetic imageNano-Banana Version
GPT-5.6 Sol24.7%18.9%
Gemini 3.8 Flash17.9%15.6%

[1] Lu, R., Ma, Y., Chen, X., Luo, L., Wu, Z., Pan, Z., Liu, X., Lin, Y., Li, H., Liu, W., Hao, Z., Gao, X., Nie, S., Wei, Y., Xie, Z., Chen, T., & Zeng, G. (2026). Thinking with visual primitives [Technical report]. DeepSeek-AI. https://www.k-a.in/Thinking_with_Visual_Primitives.pdf

[2] Chen, L., Xie, W., Liang, Y., He, H., Zhao, H., Yang, Z., Huang, Z., Wu, H., Lu, H., Charles, Y., Bao, Y., Fan, Y., Li, G., Shen, H., Chen, X., Xu, W., Si, S., Cai, Z., Chai, W., . . . Li, K. (2026). BabyVision: Visual reasoning beyond language (arXiv:2601.06521). arXiv. https://doi.org/10.48550/arXiv.2601.06521

W2.5 The sharp gap—frontier models below 20% versus 82.9% for the fine-tuned model trained on pre-style-transfer images—offers weak evidence for the benchmark's difficulty, given that the latter is essentially an in-domain evaluation.

We see that the fine-tuned Qwen3.5 results could be read as bearing on benchmark difficulty, and we would like to clarify that this was not their intended purpose. Qwen3.5-9B fine-tuning experiments are solely meant to ablate different reasoning strategies, are not intended to be evidence of difficulty or compared to zero-shot frontier model performance. These models are trained on oracle, expert traces programmatically generated using privileged information from the data construction process, so it is expected that fine-tuned models perform significantly better than frontier models. We will revise the text to make this point more clear.

W3 The conclusions are overgeneralized, as no controlled comparison is made against other grounding variants such as crop/zoom or feature replay.

Thank you for raising this point. Our intended scope is specifically textual-coordinate grounded reasoning (noted in Line 107 footnote), but we agree that the current framing may suggest broader claims. We are happy to alter the framing in the title and body to make it clear that we primarily consider textual-coordinate grounded reasoning. Additionally, our paper already contains results for crop-based models like DeepEyes. In Table 1 of our paper, we apply our online randomization techniques to DeepEyes, where we randomize the coordinate and crop. We find the model does not significantly rely on its crops for spatial reasoning or counting. (+0.02 ± 0.26 on SpatialMQA, -0.32 ± 0.28 on CountQA). We will perform more detailed experimentation and evaluation to extend these results.

We would deeply appreciate it if you could clarify whether these clarifications and proposed experiments would address your concerns.

W1 follow up All self-training experiments rely on a single backbone (Qwen3-VL-8B) with only 6,368 SFT samples.

We rerun our SFT experiments across Molmo2-8B and InternVL3.5-8B, using the same protocol described in the paper, and report the results in the table below. As demonstrated, our results generalize across multiple backbones, with counting benchmarks seeing positive lift from grounding, while spatial reasoning and perception benchmarks generally see non-significant change. Significant differences are bolded with same protocol as paper.

BenchmarknInternVL3.5-8BMolmo2-8B
Spatial reasoning
SMQA1076+0.6 ± 1.3+0.6 ± 1.7
MHS1500−0.8 ± 1.3+0.5 ± 2.0
EmbSpatial3640+0.5 ± 1.3+0.5 ± 1.2
VSR1222+0.6 ± 2.6−0.5 ± 1.1
OmniSpatial1533−0.1 ± 1.1−1.4 ± 1.3
ERQA400−0.0 ± 2.2+0.6 ± 2.7
Fine-grained perception
V*191+0.7 ± 5.6−0.4 ± 2.9
MME-RW1919−1.3 ± 1.7−0.7 ± 2.0
TreeBench405−0.2 ± 2.1+0.1 ± 2.8
Counting
CountQA1001+2.5 ± 2.0+2.4 ± 2.0
FSC-1471190+2.2 ± 1.7+1.1 ± 1.5

We report (grounded - text) performance.

W2 follow up Concerns remain about BrickBench's validity. Its programmatically generated, Nano-Banana-restyled brick-stack tasks are highly artificial and far removed from real-world visual reasoning.

As mentioned above, we believe that tasks in BrickBench are simple for humans to solve while being difficult for models, and that BrickBench provides significant value in measuring this gap. We perform a human study across 20 participants, each given 5 questions from the benchmark. We find that humans still vastly outperform models on these tasks, demonstrating the value of BrickBench.

ModelCrop IDCountingDuplicateUnpairedOverall
Humans95.895.093.392.394.1
GPT-5.6 Sol37.68.311.620.419.5
Gemini 3.8 Flash22.815.58.817.616.2
Claude Opus 530.810.78.411.615.4
Qwen 3.8 Max7.62.013.212.48.8

Note that the Crop ID numbers differ slightly from the paper table. We use more lenient grading that doesn't require picking the specific piece in the stack, and only requires picking the correct stack, which should favor models with less fine-grained pointing capabilities. We observe the models only increase in performance slightly.

W3 follow up The conclusions are overgeneralized, as no controlled comparison is made against other grounding variants such as crop/zoom or feature replay.

We reran our coordinate randomization probe across a variety of existing public models that use crop/zoom and feature replay approaches. We find that many existing models are not sensitive to randomization of their grounding, demonstrating the flexibility of our probe for other grounded reasoning variants. Significant changes are bolded.

ModelMechanismMHSSMQAV*CountQA
Crop / zoom tool
DeepEyes-7Bcrop tool+0.63 ± 0.24+0.02 ± 0.26−0.68 ± 0.38−0.32 ± 0.28
Pixel Reasoner-RL-v1crop tool−0.87 ± 0.59+0.03 ± 1.19−1.57 ± 1.59−0.03 ± 0.64
Feature replay
v1-7Bfeature replay−0.44 ± 0.50−0.53 ± 0.72+1.92 ± 1.37+0.73 ± 0.72
Text coordinates (paper Table 1)
TreeVGR-7B-CItext boxes−0.62 ± 0.33−0.50 ± 0.28+0.55 ± 0.71−0.05 ± 0.31
TreeVGR-7B-RLtext boxes−0.17 ± 0.22−0.21 ± 0.19−0.08 ± 0.35−2.85 ± 0.28
iVGR-CItext boxes−0.93 ± 0.29−0.41 ± 0.32−0.99 ± 0.51−2.68 ± 0.36
Appended (ours)text points−1.64 ± 0.31−0.61 ± 0.35−2.36 ± 0.59−0.38 ± 0.38
Reflection (ours)text points−3.63 ± 0.30−2.50 ± 0.35−4.16 ± 0.62−4.44 ± 0.37
Reviewer_pqhdOpenReview ↗

Official Comment by Reviewer

I sincerely thank the authors for their clarifications and the extensive additional experiments conducted during the rebuttal period. The multi-backbone SFT results and the extended evaluations on crop/zoom and feature-replay models have addressed my concerns regarding W1 and W3.

However, my concern about the validity of BrickBench in W2 still stands, and this aligns with Weakness 2 raised by Reviewer 72BK. While the new human study and the tests on raw 2D images show that the low performance is not just due to the Nano-Banana image style, they still do not clarify what the benchmark actually measures. As Reviewer 72BK noted, text-only fine-tuning pushes Qwen3.5-9B from 5.2% zero-shot to 61.2%, far above all frontier models which score between 14.4% and 18.9%. The authors explained in W2.5 that this fine-tuning uses oracle traces for ablation purposes. Even so, such a large jump without grounded tokens suggests that frontier models mainly struggle with out-of-distribution data or the strict point-based answer format, rather than a lack of grounded reasoning.

I will take the other reviewers' feedback into account and engage in further discussions with the AC before finalizing my overall rating.

Dear Reviewer pqhd,

Thank you again for your review. As the discussion period nears its end, we wanted to check whether our response addresses your concerns.

Reviewer_SxfcOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This work explores textual grounded reasoning (such as coordinates without accompanying visual), arguing that coordinates are useful as references to specific elements with no clear description and disambiguation possible through natural language text. The authors investigate this through a variety of experiments: [1] Coordinate corruption, where model-generated coordinates are randomized and KV-cache is rebuilt [2] Controlled training experiments across grounded and ungrounded recipes over a range of benchmarks [3] Synthetic probes. The authors find that grounding provides a statistically significant improvement on counting but not on perception or spatial reasoning.

Critical strengths
  1. This paper distinguishes producing coordinates from benefiting from them - the combination of coordinate corruption and grounded-versus-text training comparisons is more informative than evaluating task accuracy alone.
  2. The empirical analysis includes a comprehensive benchmark suite and substantial ablations. The synthetic probes add additional evidence for the usefulness of grounded reasoning both in the presence and absence of text labels.
Critical weaknesses
  1. Grounding is supervised and rewarded during training, but the authors do not systematically report whether intermediate coordinates identify the intended objects, or whether models can correctly identify visual content at supplied coordinates.
  2. "undescribable" references are not distinguished usefully from lengthy or imprecise descriptions.
Questions for authors
  1. How accurately do models produce and interpret coordinates? Could the limited gains on spatial reasoning and perception reflect a localization bottleneck? This point could benefit from reporting localization accuracy, or testing visual identification at supplied coordinates to establish a model baseline.
  2. What characterizes a reference as "undescribable"? Is there a rubric established? Does the grounding advantage persist after controlling for reference length and precision, and how does it vary with object density or ambiguity?
3 comments

Q1 How accurately do models produce and interpret coordinates? Could the limited gains on spatial reasoning and perception reflect a localization bottleneck? This point could benefit from reporting localization accuracy, or testing visual identification at supplied coordinates to establish a model baseline.

We find that models are quite strong at producing the correct coordinates. TreeBench provides ground truth boxes for grounded reasoning evaluation. We find that both existing models and our trained models perform well on producing correct coordinates, which does not suggest localization being the bottleneck.

ModelPoint-in-box recallPoint precision
Ours, RL grounded (3 seeds).747.670
Ours, SFT grounded (3 seeds).718.562
TreeVGR-7B-RL.636.678
iVGR-8B (RL).464.744

where the point-in-box metric uses the center of the predicted box for TreeVGR/iVGR. Furthermore, we find that better localization is not correlated with performance. In our training data randomization experiments (Figure 3), we actually find that as increasing percentages of coordinates in the SFT data are randomized, coordinate accuracy on TreeBench drops sharply, but performance remains flat.

% of training data coords randomized0%5%20%50%100%
TreeBench QA acc45.1046.5848.0747.6547.00
TreeBench Grounding acc (pt_recall).708.656.629.466.142

We will perform additional experiments to investigate whether visual identification of content at the coordinates is a major gap and report results in the discussion period.

Q2 What characterizes a reference as "undescribable"? Is there a rubric established? Does the grounding advantage persist after controlling for reference length and precision, and how does it vary with object density or ambiguity?

We would characterize “describable” as when an object can be uniquely identified by a short text description [1] of its properties, and “undescribable” when no such description is unique. That is, an object is undescribable when no such description is unique. We can quantify how “undescribable” an object is based on the number of objects that match the best property-based description (i.e. the level of ambiguity), and hypothesize that as the number of objects increases, the comparative value of grounding increases.

To test this, we will perform a new experiment where we vary ambiguity directly while holding images, trace template and length fixed. We will use a modified version of our shape counting task, where every shape has a label, and we vary how many shapes share a label per image (i.e. increasing/decreasing ambiguity of phrase), and compare text reasoning vs grounded reasoning performance. We will report results in the discussion period.

[1] (We note that one could discretize the image into increasingly fine cells, and say something like “cell 23,32” but we consider this a coarser version of grounding).

We would deeply appreciate it if you could clarify whether these clarifications and proposed experiments would address your concerns.

Q1 followup How accurately do models produce and interpret coordinates? Could the limited gains on spatial reasoning and perception reflect a localization bottleneck?

As mentioned, we perform an additional experiment to examine if visual interpretation of content at the coordinates is a major gap. We construct MCQ questions using TreeBench ground truth and examine if the model produces the correct answer given the text coordinates, compared to identifying the content with a red box over the area. The baseline chance accuracy is .420. Accuracy is averaged over 3 decode seeds.

Location given asGrounded SFTText SFT
Box in text.671.685
Red box drawn on the image.768.749

Although both models performs better using a visual prompt as opposed to a text coordinate prompt, they are able to generally interpret the visual content given only the text coordinate, and performs well over baseline accuracy. Combined without previous results in the previous reply, our results suggests that the limited gains on spatial reasoning are likely not due to a localization or understanding bottleneck. Instead, we hypothesize that text SFT may be sufficient to implicitly learn the same localization capabilities as grounding when objects in the image have simple references, as demonstrated in our undescribable reference probe experiments.

Q2 followup What characterizes a reference as "undescribable"? Is there a rubric established? Does the grounding advantage persist after controlling for reference length and precision, and how does it vary with object density or ambiguity?

We perform a new experiment where we vary ambiguity directly while holding images, trace template and length fixed. We use a modified version of our shape counting task, where every shape has a label, and we vary how many shapes share a label per image (i.e. increasing/decreasing ambiguity of the text phrase), and compare text reasoning vs grounded reasoning performance. We report the delta between grounded and text performance for scenes with varying amounts of ambiguity. We find that as ambiguity is increased (number of shapes sharing label), the value of grounded reasoning consistently increases, which aligns with our earlier hypothesis.

Shapes sharing each labelText CoTGrounded CoTΔ (grounded − text)
1 (unique)94.593.6−0.9
291.993.9+2.0
486.292.5+6.4
866.992.5+25.7

Dear Reviewer Sxfc,

Thank you again for your review. As the discussion period nears its end, we wanted to check whether our response addresses your concerns.

Comments on the paper

2 comments

Dear Reviewers and Area Chair,

Thank you all for your thoughtful feedback. We are grateful for general reviewer agreement on the informativeness of our contributions: pqhd and Sxfc report gaining "substantially deeper understanding of the problem," while 72BK note that "the finding … is interesting and valuable." In each reviewer thread, we have provided initial clarifications and welcome any further questions or requests for analyses.

These are not our full rebuttals; it is an early post with results we already have and is intended to confirm what additional experiments or clarifications would be most helpful, especially since some experiments might take longer to run. We would be grateful if reviewers could review the clarifications and see if they address their major concerns. If not, we are happy to discuss further.

Dear Area Chair and Reviewers,

We sincerely thank you for your valuable time and efforts. We have provided our follow-up rebuttal, with the additional experiments proposed in the initial replies. Below, we summarize the main merits and concerns raised by the reviewers, along with our responses to these concerns during the discussion period. For detailed experimental results, please see the corresponding Reviewer Response thread.

We make several key contributions in our work

  • A training-free probe technique to measure grounding usage
  • Detailed experimental analysis of a grounding training recipe, which finds that grounding is primarily useful for counting and tasks requiring tracking undescribable references.
  • BrickBench, a new benchmark that requires tracking undescribable references

We appreciate that the reviews found strength in our experimental analysis for both our training-free probes and our training experiments. Sxfc finds “comprehensive benchmark suite and substantial ablations”, pqhd notes that our “probes cleanly isolate reference describability as the key variable, and the performance reversal is compelling evidence,” and 72BK finds our results on grounding vs text reasoning performance “interesting and valuable, and it is well-supported.” 72BK finds that "BrickBench is novel and could be important for the community."

To address concerns about generalizability to other grounded reasoning approaches and across different models, we have performed several new experiments, described below. We will clarify in our paper more explicitly that the primary scope of our paper is on textually grounded reasoning.

  • We have verified our results across different VLM backbones
  • We have performed our probe tasks for crop/zoom variants of models, and find that our results hold across different grounded reasoning variants.
  • We have performed our decode-randomization analysis across a wider variety of models, including feature replay and crop/zoom grounded reasoning models.

To address concerns about BrickBench, we have provided experimental results across different style variations, which show that the benchmark remains difficult for both 2D render and Nano-Bannana style distributions. We have also conducted a new human study that shows that humans are able to perform significantly better than existing models on BrickBench. We note that while models (e.g. GPT 5.6-Sol) are able to perform well (>70%) on popular synthetic benchmarks such as BabyVision, they still perform poorly (<20%) on BrickBench tasks. We will also clarify that the Qwen3.5 experiments are solely meant to represent ablation results over different reasoning strategies. We also find that point-based answer formats are not the primary bottleneck for model performance, as our counting based questions also see low performance.

Finally, we have performed a variety of other experiments to further clarify and augment our contributions, such as our analysis of interpreting coordinates, and also our controlled experiments on reference ambiguity. Full details are provided in the reply threads. If there are remaining concerns, please feel free to let us know. Thank you all again for helping us improve and strengthen our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.