- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The authors introduce OmniGrounding, a large-scale native-camera 3D grounding dataset. The authors also propose OmniGrounder, which injects geometric features into VLM to support 3D grounding. Experiments on five benchmarks demonstrate the effectiveness of the proposed design.
Critical strengths
- The proposed large-scale dataset combines multiple domains, and could be valuable for training generalizable 3D grounding models.
- The improvement in focal-length estimation and 3D grounding suggest that explicit geometric features can benefit VLM-based model for 3D grounding.
Critical weaknesses
- The novelty of the proposed dataset is not clear. The authors define native-camera 3D grounding as "the predicted box is expressed in the coordinate frame of the camera that captured the input" (line 032-033). However, existing datasets, such as WildDet3D, already support camera-frame 3D boxes. The Tab. 6 in the appendix distinguishes the existing datasets, but its unclear why WildDet3D is not native-frame. Its annotations already provide camera-frame 3D boxes. The proposed Omnigrounding dataset also reuses several existing 3D datasets. The benefits of the new data construction and annotation alignment are not fully demonstrated.
- The experiment section is weak. First, Tab. 1 mainly compares with general-purpose VLMs. The authors should also consider fine-tuning some open-source 3D grounding specialists, such as WildDet3D and DetAny3D, using the proposed data. Currently it is not clear how much of the improvements comes from the new training data and model design separately. Second, the indoor benchmarks use mean per-image [email protected] (line 358-359) instead of standard 3D detection metrics (such as mAP). The recall does not measure precision, and may not adequately reflect the grounding capability. Finally, ScanRefer seems to be the only referring expression grounding benchmark, and the performance seems very bad (5.89% [email protected] in Tab. 1). This questions the data quality of the referring expressions in Omnigrounding, and also the effectiveness of the proposed design on the referring expression grounding task.
Questions for authors
- Based on weakness 1, could the authors further clarify the differences between the proposed native-camera 3D grounding task and the existing data formulation (such as Wilddet3D dataset)? Why is WildDet3D marked as not supporting native-frame 3D boxes in Tab. 6?
- Based on weakness 2, could the authors compare against specialized 3D grounding models, such as WildDet3D and DetAny3D? Fine-tuning these models using the proposed dataset would help evaluate whether the improvements come from the dataset or the model design.
- Based on weakness 2, why do the authors report mean per-image [email protected] for the indoor detection benchmarks? Could the authors additionally report standard 3D detection metrics, such as mAP at different IoU thresholds?
- Based on weakness 2, could the authors provide further analysis of the failure cases for ScanRefer referring expression grounding? It would also be useful to evaluate on additional referring-expression grounding benchmarks.
- In terms of the annotation quality, how is the metric scale of game-video annotations validated? Could the authors provide some quantitative measurements?
- Could the authors provide a computational cost analysis, such as inference latency, and compare against VLM baselines and additional grounding specialists?
3 comments
Referring-Expression Grounding and Metric Scale (2/3)
Q4: Referring-expression grounding
"ScanRefer seems to be the only referring expression grounding benchmark, and the performance seems very bad (5.89% [email protected] in Tab. 1) ... could the authors provide further analysis of the failure cases ... It would also be useful to evaluate on additional referring-expression grounding benchmarks."
ScanRefer failure analysis. Our ScanRefer evaluation uses RGB frames and the official referring expressions, rather than a reconstructed 3D scene. Under the Table 1 protocol, providing the ground-truth focal length raises [email protected] from 5.89 to 13.62 (+7.73 percentage points; Table R2), identifying focal-length estimation as one contributor to the low score.
Table R2. ScanRefer [email protected] under the Table 1 protocol.
| Model | ScanRefer [email protected] |
|---|---|
| Baseline | 0.79 |
| OmniGrounder, RGB only | 5.89 |
| OmniGrounder, RGB + GT focal length | 13.62 |
To analyze the remaining failures, we examined a separate post-submission evaluation using an SFT checkpoint and up to 48 input frames, covering 4,931 queries from 84 scenes whose scene IDs do not occur in our fine-tuning data. [email protected] is 5.62 with predicted focal length and 12.35 with reference focal length.
-
Non-overlapping predictions. Among the missed queries, 80.4% of the predicted boxes have zero overlap with the target, while 19.6% overlap it below the 0.25 threshold.
-
Same-category distractors. [email protected] is 16.46 on the unique subset and 6.64 on the multiple subset, which contains same-category distractors. However, 53.26% of predictions on the unique subset still have zero overlap with the target, indicating that same-category instance confusion explains only part of the failures.
These results show that focal-length error and referent ambiguity contribute to the gap, while substantial target-localization errors remain even with reference focal length. In OmniGrounding, referring queries cover six attribute groups, and each query–object correspondence is verified during data construction (Section 3.3).
Additional benchmark. We further evaluate on Multi3DRefer and report [email protected] (Table R3). OmniGrounder achieves 13.63, exceeding the baseline by 11.73 percentage points and RoboBrain 2.5-8B, the strongest of the listed VLMs, by 5.00 percentage points.
Table R3. Multi3DRefer, [email protected].
| Model | [email protected] (%) |
|---|---|
| Baseline | 1.9 |
| Pelican-VL-7B | 0.29 |
| MIMO-Embodied-8B | 7.5 |
| RoboBrain 2.5-8B | 8.63 |
| ACE-Brain-0-8B | 7.6 |
| RynnBrain-8B | 1.42 |
| OmniGrounder | 13.63 |
Q5: Validation of the metric scale of game-video annotations
"... how is the metric scale of game-video annotations validated? Could the authors provide some quantitative measurements?"
We performed a quantitative size-consistency check after Pi3X-based scale recovery, comparing the game-video box dimensions with those of the corresponding categories in real-sensor datasets: SUN RGB-D, ARKitScenes, nuScenes, KITTI, and Objectron.
-
Shared categories. Across 14 categories with at least 20 game boxes and 100 real boxes, covering 3,522 game boxes from 229 scenes, the median game-to-real size ratio is 1.00, with a 95% scene-bootstrap interval of [0.98, 1.13]. Category-level medians range from 0.63 to 1.60.
-
Cars (99 boxes). Median height, length, and width are 1.32, 4.77, and 1.84 m, compared with 1.67, 4.54, and 1.91 m in nuScenes and KITTI.
-
Standing persons (439 boxes). For person boxes whose height is at least 2.5 times the longer footprint dimension, the median height is 1.63 m, compared with 1.79 m for nuScenes and KITTI pedestrians.
These measurements provide quantitative evidence that the recovered dimensions are broadly consistent with real-world object sizes, while also documenting category-level variation. They assess metric-scale plausibility at the distribution level, rather than per-instance accuracy against engine-provided metric ground truth. Box-level quality of the same source is audited separately: on a 100-image sample, human review retains 307 of the 328 automatically accepted boxes, 295 of them without modification.
Inference Latency and Computational Cost (3/3)
Q6: Computational cost
"Could the authors provide a computational cost analysis, such as inference latency, and compare against VLM baselines and additional grounding specialists?"
We measured single-image inference latency on one NVIDIA H200 with batch size 1. Every model answers the same category-level 3D detection queries on 10 images, two from each of SUN RGB-D, ARKitScenes, Hypersim, KITTI, and nuScenes, with one warm-up call and three timed calls per image. The reported time is that of normally completed calls; it covers reasoning, generation, and output parsing, and excludes model loading. DetAny3D's time includes its Grounding DINO proposals.
Table R4. Single-image inference latency.
| Model | Latency range (s) | Mean (s) |
|---|---|---|
| WildDet3D | 0.48–1.31 | 0.74 |
| DetAny3D | 0.59–1.64 | 1.06 |
| Qwen3-VL-8B-Thinking | 2.62–5.69 | 4.02 |
| Qwen3.5-9B | 5.98–48.38 | 13.7 |
| OmniGrounder | 3.09–7.68 | 4.23 |
OmniGrounder takes 4.23 s per query on average, comparable to Qwen3-VL-8B-Thinking (4.02 s) and offering a 3.2× speedup over Qwen3.5-9B (13.70 s), the backbone it builds on. The two specialists answer in about 1 s or less, since they predict boxes without autoregressive decoding, whereas the VLMs generate boxes token by token.
OmniGrounder’s reasoning trace is specific to grounding: it commits to the focal length and coarse object positions before emitting the boxes. Its outputs have a median of 108 tokens, compared with 170 for Qwen3-VL-8B-Thinking and 312 for Qwen3.5-9B. This strengthens 3D grounding while retaining the LLM-based interface for embodied reasoning and action planning; with asynchronous integration, grounding inference can remain outside the high-frequency action-control loop.
Dataset Novelty, Specialists, and Detection Metrics (1/3)
Thank you for the careful review and for catching the error in Table 6. We respond to each weakness together with the questions that follow from it.
W1 & Q1: Dataset novelty and the WildDet3D entry in Table 6
The Native-frame entry for WildDet3D in Table 6 is an error: WildDet3D provides camera-frame 3D boxes, and we will correct it.
The coordinate convention is shared with WildDet3D. OmniGrounding differs in supervision: WildDet3D-Data prompts with category names on single images, whereas OmniGrounding adds instance-level referring queries, multi-frame clips, and focal length as a generation target. It also aligns box conventions across sources, refines 1,445,349 inherited boxes, and adds open-vocabulary boxes from game videos.
Replacing the WildDet3D training data (Omni3D + WildDet3D-Data) with OmniGrounding improves [email protected] for all five models on all three benchmarks, by 4.04–9.59 points (Table R1).
Table R1. [email protected] under two training-data settings. Each cell: trained on Omni3D + WildDet3D-Data → trained on OmniGrounding. DetAny3D receives its box prompts from Grounding DINO given the text query.
W2 & Q2: Specialist comparison, data versus model
Table R1 includes the WildDet3D and DetAny3D specialists trained on OmniGrounding and separates the two factors.
Data. With the model fixed, OmniGrounding raises [email protected] for every model, including WildDet3D (+5.34 / +4.04 / +5.59) and DetAny3D (+5.25 / +5.06 / +5.28) on SUN RGB-D / ARKitScenes / Hypersim.
Model. With the data fixed to OmniGrounding, OmniGrounder exceeds its Qwen3.5-9B backbone by 11.89 / 7.29 / 9.89 and the strongest specialist, WildDet3D, by 2.40 / 4.03 / 3.44.
Q3: Detection metric
We initially chose mean per-image [email protected] for Table 1 primarily to measure target coverage under category-conditioned queries, giving each image equal weight. However, this choice did not account for false-positive predictions. We therefore additionally report [email protected] for all models in Table R1. The IoU threshold of 0.15 is consistent with prior evaluations of VLM-based 3D box prediction, which also report average precision at this threshold [1–4]. OmniGrounder achieves the highest AP on all three benchmarks under both training-data settings.
[1] Shuai Bai et al. Qwen3-VL Technical Report. arXiv:2511.21631, 2025.
[2] Gemini Robotics Team et al. Gemini Robotics: Bringing AI into the Physical World. arXiv:2503.20020, 2025.
[3] Rui Yang et al. Visual Spatial Tuning. arXiv:2511.05491, 2025.
[4] An-Chieh Cheng et al. Grounded 3D-Aware Spatial Vision-Language Modeling. arXiv:2605.30307, 2026.