acceptodds
Withdrawn from ICLR 2027

GroundBricks: A Configurable Benchmark for Compositional Visual Grounding

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Visual grounding aims to localize image regions specified by language queries. Existing benchmarks, however, often focus on direct query–object matching, providing limited and systematic control over the diversity and complexity of grounding tasks. Constructing richer benchmarks typically requires additional region-level annotations, which are costly and difficult to scale. We argue that the required localization supervision already exists. Current object detection and instance segmentation datasets contain abundant verified region annotations. What is missing is a mechanism for composing these annotations into diverse and controllable grounding tasks. Based on this insight, we introduce GroundBricks, a configurable benchmark for compositional visual grounding, together with a fully automated construction pipeline. GroundBricks covers five capability categories: referential grounding, set grounding, attribute reading, relational chaining, and knowledge-grounded selection. It also spans three complexity levels. Our pipeline uses existing object detection annotations as its foundation, enriching annotated regions into structured semantic representations that serve as reusable grounding bricks for composing different tasks. The pipeline then applies rule-based query generation and validation. This process requires neither manually written queries nor additional region annotations. GroundBricks provides a fixed test set for reproducible comparison while supporting extensions through new composition rules and query templates. Its evaluation scope can therefore evolve alongside advances in model capabilities.

Primary area:
datasets and benchmarks
Keywords:
Visual Grounding, Compositional Reasoning, Multimodal Learning, Automated Dataset Construction, Configurable Benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_cx9tOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes generating diverse compositional grounding tasks by reusing existing grounding annotations, automatically extracting attributes and spatial relations, and combining them with predefined rules. Experimental results demonstrate that the benchmark effectively enhances the performance of Multimodal Large Language Models (MLLMs) on complex grounding tasks.

Critical strengths

The introduction to the benchmark is highly detailed. The authors successfully constructed reliable training and evaluation sets under low-resource constraints, which appears to be effective for MLLM training.

Critical weaknesses

The paper positions GroundBricks as a benchmark for compositional visual grounding; however, the main experiments focus primarily on its training benefits. There is a lack of sufficient analysis regarding the benchmark's own evaluation validity, discriminability, and annotation reliability. Furthermore, since the training and test sets share the same task construction mechanism, the evaluation results may be biased by the template distribution.

  1. Unclear contribution: Previous works have already explored compositional task generation based on structured scene representations, such as CLEVR-Ref+, Cops-Ref, and Ref-Reasoning. This paper lacks a direct comparison between GroundBricks and these methods to demonstrate the former's unique advantages and the latter's specific limitations in terms of scalability, task coverage, or difficulty control. Consequently, the current contribution appears to be more of an integration of existing techniques rather than a fundamental contribution to the task itself.
  2. Lack of reliability verification: The entire dataset relies on a single Vision-Language Model (VLM), which inevitably introduces the large model's systematic biases into the dataset construction. Similarly, all grounding annotations are limited to the existing Qwen model, lacking validation on baselines such as LocateAnything or GroundingDINO. Moreover, the dataset construction lacks human quality assessment, making it difficult to confirm the reliability of the dataset.
  3. Evaluation of difficulty: The paper categorizes tasks into L1, L2, and L3 primarily based on the structural complexity of the queries. However, structural complexity does not necessarily directly reflect the actual reasoning difficulty, as model performance can also be influenced by factors such as target saliency, visual ambiguity, and task categories. Additionally, the task types included in different complexity levels are not entirely consistent; therefore, the performance differences observed in Figure 3 may be influenced by the task distribution rather than solely stemming from the depth of compositional reasoning.
  4. Potential distribution shift: The overall dataset filtering process may retain visually salient targets, leading to a distribution shift in the data. This is unfavorable for models when handling actual grounding tasks in real-world open scenarios. Typos: Line 249: "the resulting hierarchy are" (subject-verb agreement error). There is an inconsistency in the statistics of the attribute vocabulary size between Table 5 and Table 6 on Page 17.
Questions for authors

Has the quality of the GroundBricks test set been independently evaluated through human annotation? If so, could the authors report the accuracy of the generated attributes, spatial relations, and grounding answers?

What specific evaluation capabilities does GroundBricks provide that cannot be adequately assessed by existing compositional grounding benchmarks such as CLEVR-Ref+, Cops-Ref, and Ref-Reasoning?

Reviewer_nJfGOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

GroundBricks is a configurable benchmark and training dataset for compositional visual grounding. It builds structured scene representations from existing region annotations, then uses explicit selectors and validation rules to generate tasks with different reasoning demands and answer formats. The experiments assess transfer to external grounding benchmarks and compare generated supervision with raw-annotation training. An internal evaluation examines performance across capabilities and difficulty levels.

Critical strengths
  1. The task definitions are explicit and extensible, with construction rules that check uniqueness and target-set consistency.

  2. The generated supervision transfers beyond the new benchmark, with the clearest benefits on spatial and fine-grained grounding.

Critical weaknesses
  1. The raw-data control mixes natural images with GUI and OCR data, whereas GroundBricks uses natural images only and changes the query and answer distributions. Matching the number of training examples does not control for these differences, so the extra gain cannot be attributed specifically to composition.

  2. The scoring rules do not say how a complete object set or a box–text answer is judged correct. In particular, it is unclear whether missing an object or returning the right box with the wrong text counts as failure; these choices directly affect the set-grounding and relational-chaining scores.

Questions for authors
  1. Can you run a small control comparing simple and composed queries from the same natural-image subset, using the same enriched instance annotations? Please keep the answer format, number of examples, and training steps fixed so that query composition is the main difference.

  2. Please state the complete-answer scoring rule for multi-object and box–text queries. How are a missing object, an extra box, and a correct box paired with incorrect text handled, and does the order of predicted objects matter?

  3. After the existing construction checks, what errors remain in attribute labels, knowledge-based selections, and target-set completeness? A small independent annotation audit would help, since the same Qwen model performs both annotation and final verification.

  4. Does the difficulty trend hold within referential grounding or set grounding alone? The aggregate levels mix different capabilities: attribute reading is absent from the hardest level, while relational chaining appears only there.

Reviewer_BgTPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces GroundBricks (GB), a configurable benchmark for compositional visual grounding built from existing detection/segmentation annotations. The core insight is that localization supervision already abundantly exists—what's missing is composability. The authors design a five-stage automated pipeline that enriches bounding-box annotations with structured attributes and four-directional spatial relations, forming reusable "visual bricks" composed into 21 task types across 5 capability categories, 3 difficulty levels, and 2 modifiers (abstention/multi-image). The pipeline separates model-based annotation (Qwen3.8-27B) from deterministic rule-based validation, including eight structural admission gates. The authors fine-tune five models (4B–35B) and evaluate on 10 external benchmarks, with three base models achieving average gains of +13.9, +7.5, and +8.4 points. Contributions include: the brick-based construction framework, the GroundBricks dataset, and systematic evaluation.

Critical strengths
  1. Well-motivated core insight. The observation that existing detection/segmentation annotations already contain the localization supervision needed for grounding tasks, and that the missing piece is composability rather than supervision itself, is elegant and practical. Reusing trusted bounding-box annotations as the spatial foundation, while adding structured semantic enrichment on top, is a sound design principle that sidesteps the noise issues of fully model-generated grounding data.

  2. Comprehensive evaluation. The paper evaluates on five models spanning 4B–35B parameters and ten external benchmarks covering box-level, point-level, and answer-level evaluation. The breadth of downstream evaluation is commendable and provides convincing evidence of transfer across diverse grounding tasks.

  3. Controlled comparison design. Table 3 is a well-conceived experiment: matching training data volume (145,260 samples), using the same underlying annotations, and comparing raw annotations vs. GroundBricks composition. This is the right experiment to isolate composition from scale. The fact that GroundBricks uses only natural images while raw annotations also cover GUI and OCR makes the comparison conservative in the right direction. Figure 4's decomposition across five capabilities reveals differential scaling behavior: Chain reasoning improves 14.4 points from 4B→35B while Set grounding does not improve consistently. This fine-grained analysis is informative and goes beyond a single aggregate score—a genuine strength of the benchmark's structure.

Critical weaknesses
  1. Controlled comparison gains are marginal and include notable regressions. The headline +0.76 average gain in Table 3 masks substantial per-benchmark variation. GroundBricks decreases on RefCOCO (−3.23), PIO (−2.80), OSWorld-G (−1.74), and TreeBench (−5.12). The paper frames this as "composition improves performance under matched training scale," but the average gain is within the range of training noise for individual benchmark evaluation. The claim that "the improvement does not come from data scale alone" (Section 1) is technically supported but the effect size is underwhelming—one could equally conclude that composition trades losses on direct grounding for gains on compositional reasoning. The paper should discuss this trade-off explicitly rather than presenting the +0.76 as a uniformly positive result.
  2. Non-base model gains are negligible. Qwen3.8-27B gains only +0.03 average points (60.49→60.52, Table 2). This is essentially statistical noise, yet the paper claims "Non-base models also benefit." For Qwen3.5-4B-Instruct, the +1.95 average gain comes with sizable regressions on RefCOCO (−10.83), RefSp(-5.41). The paper should be more honest about the boundary of this benefit: already-instructed models with strong grounding capabilities show minimal and sometimes negative transfer, which limits the practical utility of GroundBricks as a training data source.
  3. Limited spatial relation types. Only four directional relations (left, right, above, below) are used. Real-world spatial language involves far richer relations: "near", "between", "surrounding", "in front of", "behind", "next to", "at the edge of". The paper does not discuss this limitation or whether the framework can accommodate additional relation types. This restricts the benchmark's coverage of spatial reasoning, which is particularly relevant given that "relational chaining" is one of the five capability categories.
  4. No human validation of attribute annotation quality. The entire pipeline hinges on the reliability of Qwen3.8-27B's attribute annotations, which serve as ground truth for all downstream tasks. Yet the paper provides no systematic human accuracy assessment of these annotations. The Ethics statement acknowledges "automated attribute extraction may introduce semantic biases or errors" but the severity is neither quantified nor bounded. Even a small-scale human audit (e.g., 200 instances) would substantially strengthen confidence in the benchmark. Without this, the "ground truth" in GroundBricks rests on an unverified foundation.
Questions for authors

See weaknesses above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.