- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes generating diverse compositional grounding tasks by reusing existing grounding annotations, automatically extracting attributes and spatial relations, and combining them with predefined rules. Experimental results demonstrate that the benchmark effectively enhances the performance of Multimodal Large Language Models (MLLMs) on complex grounding tasks.
Critical strengths
The introduction to the benchmark is highly detailed. The authors successfully constructed reliable training and evaluation sets under low-resource constraints, which appears to be effective for MLLM training.
Critical weaknesses
The paper positions GroundBricks as a benchmark for compositional visual grounding; however, the main experiments focus primarily on its training benefits. There is a lack of sufficient analysis regarding the benchmark's own evaluation validity, discriminability, and annotation reliability. Furthermore, since the training and test sets share the same task construction mechanism, the evaluation results may be biased by the template distribution.
- Unclear contribution: Previous works have already explored compositional task generation based on structured scene representations, such as CLEVR-Ref+, Cops-Ref, and Ref-Reasoning. This paper lacks a direct comparison between GroundBricks and these methods to demonstrate the former's unique advantages and the latter's specific limitations in terms of scalability, task coverage, or difficulty control. Consequently, the current contribution appears to be more of an integration of existing techniques rather than a fundamental contribution to the task itself.
- Lack of reliability verification: The entire dataset relies on a single Vision-Language Model (VLM), which inevitably introduces the large model's systematic biases into the dataset construction. Similarly, all grounding annotations are limited to the existing Qwen model, lacking validation on baselines such as LocateAnything or GroundingDINO. Moreover, the dataset construction lacks human quality assessment, making it difficult to confirm the reliability of the dataset.
- Evaluation of difficulty: The paper categorizes tasks into L1, L2, and L3 primarily based on the structural complexity of the queries. However, structural complexity does not necessarily directly reflect the actual reasoning difficulty, as model performance can also be influenced by factors such as target saliency, visual ambiguity, and task categories. Additionally, the task types included in different complexity levels are not entirely consistent; therefore, the performance differences observed in Figure 3 may be influenced by the task distribution rather than solely stemming from the depth of compositional reasoning.
- Potential distribution shift: The overall dataset filtering process may retain visually salient targets, leading to a distribution shift in the data. This is unfavorable for models when handling actual grounding tasks in real-world open scenarios. Typos: Line 249: "the resulting hierarchy are" (subject-verb agreement error). There is an inconsistency in the statistics of the attribute vocabulary size between Table 5 and Table 6 on Page 17.
Questions for authors
Has the quality of the GroundBricks test set been independently evaluated through human annotation? If so, could the authors report the accuracy of the generated attributes, spatial relations, and grounding answers?
What specific evaluation capabilities does GroundBricks provide that cannot be adequately assessed by existing compositional grounding benchmarks such as CLEVR-Ref+, Cops-Ref, and Ref-Reasoning?