acceptodds
Withdrawn from ICLR 2027

GroundBench: Multi-Resolution Polygon Grounding Exposes the Geometry Gap in Vision-Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

A bounding box can locate the right object without describing its shape. This leaves an important question unanswered by box-based grounding scores: can a general-purpose vision–language model turn the same visual and linguistic evidence into a usable object region? We introduce GroundBench, a benchmark that asks models to express referred objects as polygons through their text-generation interface. It pairs 1,500 image–expression–referent triples across five vertex budgets, holding the questions fixed while varying output complexity. Evaluation reports filled-region overlap, coordinate-sequence completion, and polygon legality separately. Across four hosted models, more vertices do not reliably improve the output: performance degrades at the largest budget despite more faithful reference contours. Target preference can also remain high while conditional region quality is poor, showing that favouring the right object does not ensure a useful geometric description. Controlled interventions further reveal sensitivity to reasoning settings and misleading spatial language. These findings expose a geometric-output gap: locating an object does not ensure reliable explicit geometry. They identify improvement targets in region construction and output completion that a box score alone does not expose. GroundBench evaluates their combined outcome rather than latent boundary perception in isolation.

Primary area:
datasets and benchmarks
Keywords:
visual grounding, vision-language models, polygon grounding, object localization, benchmark, geometric reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_kGquOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies the problem that if we ask the VLMs to output a more precise format of the visual grounding bounding boxes, i.e., polygons at different resolutions, how the VLMs will perform. To this end, the paper constructs a benchmark, named GroundBench, to answer the question. The benchmark consists of 1500 image–expression–referent triples, and the samples are from the RefCOCO benchmark. The paper indicates that even if we allow the models to output more points, the performance is not getting better. The authors claim that this study shows even though the VLMs can locate the objects with boxes quite accurately, this does not ensure reliable explicit geometry.

Critical strengths

The ability of visual grounding for VLMs is an essential and important task to study. It is good to explore other more fine-grained formats in additional to the bounding box formats for the visual grounding task.

Critical weaknesses
  1. Concern about the major discovery and conclusion of the paper. For a general VLM that has never been trained to output the visual grounding format in a polygon with many points, it is expected that they cannot perform very well for it. As the authors have also realised, if they ask the VLM to output mask first and then convert the mask into polygon, the performance gets improved. Therefore, the major evaluation on the benchmark / protocol the authors proposed may not be a good reflection of the actual fine-grained spatial visual grounding capability of the VLMs.

  2. The paper is hard to read. It is difficult to understand the paper given the expressions and descriptions of the current submission. It would be better if the authors can polish the writing and express the ideas more clearly.

Questions for authors

I would suggest the authors to address my concerns as in the weakness section during the rebuttal.

Reviewer_XbTUOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces GroundBench, which evaluates direct polygon-coordinate generation by four general-purpose hosted VLMs on 1,500 fixed image–expression–referent triples at five exact vertex budgets: N ∈ {8, 16, 24, 32, 64}. Reference polygons are constructed deterministically from the same selected source contour. Evaluation separates fixed-denominator filled-region IoU, parseable-only IoU, and polygon legality, with additional diagnostics for relative target preference and conditional region quality.

All four main configurations have lower fixed IoU at N=64 than at N=16. The paper further examines predicted rectangles, stable-boundary subsets, thinking settings, conflicting cues, and alternative output interfaces to distinguish region quality from output reliability.

Critical strengths
  1. Holding the image, expression, and referent fixed while varying N reduces confounding from differences in question composition. Unparseable outputs receive zero under fixed-denominator scoring, while parseable-only and legality-enforced metrics distinguish region quality from reliable delivery of a legal polygon. Stable-boundary comparisons further show losses from N=32 to N=64 even when additional vertices only subdivide existing source edges, supporting the relevance of output requirements without isolating an internal mechanism.

  2. The paper includes useful controls and explicit limits on attribution. Predicted rectangles and direct polygons are compared against the same contour targets. The thinking comparison also examines items parseable under both configurations, so its gains cannot be explained solely by increased completion. Alternate references, target starting phases, geometry backends, alternative interfaces, and a limited independent repeat broaden the analysis. UID-level resampling preserves dependence across vertex budgets.

Critical weaknesses
  1. The row-level audit is incomplete for the leading Gemini configuration, limiting independent verification of its full-cohort strict results. Appendix G.2 states that strict results for Doubao, Kimi, and both Qwen configurations are recomputed from 1,500 selected raw responses per budget, whereas Gemini retains the manuscript reporting values. The historical Gemini diagnostic export contains only 500 items per budget and cannot validate the full-cohort strict results or matched strict direct-versus-box comparisons. Kimi’s stage-0 strict aggregate is another explicitly disclosed exception because a raw-response prefix is unavailable. These provenance distinctions should remain clear wherever the corresponding conclusions are presented.

  2. The mask-prediction provenance of the mask-to-polygon comparison remains insufficiently specified. Section 4.3 and Appendix F.3/Table S30 report substantially better high-budget results for this pipeline, including fixed IoU of 63.63 and legal coverage of 92.9% at N=64. However, F.3 does not clearly identify the mask predictor or connect it to the separately documented PolyFormer-L calibration. Pre-conversion mask IoU is also explicitly unreported. This limits reproduction and prevents decomposition of the results into mask-prediction quality and conversion losses, although the paper appropriately treats the comparison as one between complete pipelines rather than an isolated capability intervention.

Questions for authors
  1. Could you provide a consistent per-UID audit for Gemini’s full-cohort strict results and clarify their relationship to the historical 500-item-per-budget diagnostic export? Please identify which entries in Tables S37–S40 are recomputed by the current validator and which retain archived aggregates, with provenance for the matched strict direct-versus-box comparison. If complete responses cannot be recovered, a separately reported common-UID audit would help, while retaining and clearly qualifying the historical full-cohort summaries. Please similarly delimit the unrecomputed Kimi stage-0 strict value.

  2. Which system produces the masks in Table S30, and how does it relate to the PolyFormer-L calibration in Table S8? Please specify the predictor/checkpoint, inputs, target binding, auxiliary information, failure and filtering rules, and conversion algorithm. The manuscript states that the F.3 converter does not use ground truth, but the prediction-stage provenance remains unclear. Per-UID masks and final polygons, pre-conversion mask scores, and conversion-failure diagnostics would make the comparison more verifiable and clarify why its results differ from Table S8.

Reviewer_eAvxOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper looks at a gap in how referring-expression grounding is usually evaluated, since bounding boxes tell us whether a model found the right object but say very little about whether it can actually describe the object’s shape. The authors introduce GROUNDBENCH, which asks general-purpose VLMs to output the referred object as a polygon directly through text, with an exact number of vertices requested. They take triples and keep them fixed, and only vary the vertex budget, so any change in performance comes from the output complexity rather than the question structure themselves. Evaluation is split into region overlap, whether the model completes the coordinate list at all, and whether the polygon is geometrically legal. Their main finding is that more vertices do not lead to better regions, and performance drops at the largest budget even though the reference contour barely gets more detailed. They also show that a model can prefer the correct object over distractors and still describe its region poorly. Controlled interventions on Qwen and Gemini suggest that thinking mode helps both completion and region quality, and that misleading spatial language hurts much more than misleading colour. Overall they argue that locating an object and producing reliable explicit geometry for it are separate abilities, and that box-based scores hide this.

Critical strengths

These are the strengths of the paper:

  1. The primary research question about wanting to understand whether the model can recognize the shape of the object it detects, and not just the location of the object in the scene without a segmentation mask, seemed acceptable to me. They remove confounders and build a controlled benchmark to test only the effect of varying the number of points in the shape mask. Moreover, region overlap, completion of the coordinate list and polygon legality are reported separately, so one can see whether a model drew a bad region or never produced a usable polygon at all.

  2. The controlled interventions on the input are informative in my opinion. Removing the image causes a much larger drop than halving its resolution or turning it greyscale, so the models are clearly using the visual evidence rather than relying only on language priors. Moreover, adding a false spatial cue hurts far more than a false colour cue, even though the original expression still points to the right object. Essentially, misleading language can override what the model sees, which suggests that robustness to contradictory instructions should be evaluated alongside clean accuracy.

Critical weaknesses
  1. I find the whole "geometry gap" framing promise to be over-extended and over-claiming. When a model does produce a parseable answer, its region quality changes much less across budgets, and the drop at the largest budget comes mostly from outputs that cannot be parsed at all. The authors themselves say in the conclusion that this is about output reliability rather than an inability to perceive shape, but the title and introduction suggest otherwise.

  2. The cross-model comparisons are not done under one consistent setup. Gemini gets an extra retry in some analyses, its strict scores are not recomputed from the raw outputs, and several of its diagnostics use only a third of the questions. Output limits and temperatures also differ a lot across models, and the paper itself does not rule out truncation at the largest budget. In my opinion these comparisons should be rerun under a uniform protocol. The paper should also state which model produces the masks in the mask-to-polygon comparison, and report the specialist’s mask quality before conversion, since otherwise its low score is hard to interpret.

  3. I did not like the writing style of the paper, while the authors do state that it was used to polish the manuscript, it feels like a lot of the writing was generated by AI, especially with the use of constant scope declarations.

Questions for authors

I had a few questions to ask for clarification:

  1. At the largest budget, how many unparseable outputs were cut off by the output limit, and how many were complete but malformed, especially for Doubao, whose output limit is much smaller than the others’? Could the authors also recompute Gemini’s strict scores on all questions and report the analyses that currently use the best-effort view, such as target preference, under the same retry policy as the headline table?

  2. Which model produces the masks in the mask-to-polygon comparison, and what is its mask quality before conversion? The same question applies to PolyFormer. I believe its the main evidence that the bottleneck is producing the output rather than predicting the region.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.