acceptodds
Desk rejected at ICLR 2027

OmniGrounder: Native-Camera 3D Grounding from Diverse Data and Geometric Priors

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Embodied AI is pushing vision-language models from naming what they see toward locating it, and the core capability behind it is 3D grounding: mapping an image and a text query to an oriented 3D box that specifies an object’s location, size, and orientation. However, an object’s apparent size depends jointly on its metric size, depth, and camera focal length, making it difficult to recover metric 3D boxes across cameras. In this work, we present OmniGrounding, a native-camera dataset, and OmniGrounder, a vision-language grounding model for this task. The OmniGrounding aligns camera annotations, oriented 3D boxes, and language instructions across multiple domains with a newly proposed game-video source, providing 4.5M bounding boxes with 3M question–answer pairs for detection and referring. The OmniGrounder injects geometric priors from pretrained 3D foundation model and learns camera-discriminative features by focal-length estimation. In the mid-training ablation, geometry injection reduces focal-length mean absolute error (MAE) by 23%, from 62.3 px to 47.9 px, compared with the same-backbone baseline. Without training on WildDet3D, our full model achieves 31.48 AP, outperforming the baseline (20.16 AP) significantly. Our dataset, model, and pretrained weights will be publicly available.

Primary area:
foundation or frontier models, including LLMs
Keywords:
3D Grounding, VLM for 3D Grounding

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_juodOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The authors introduce OmniGrounding, a large-scale native-camera 3D grounding dataset. The authors also propose OmniGrounder, which injects geometric features into VLM to support 3D grounding. Experiments on five benchmarks demonstrate the effectiveness of the proposed design.

Critical strengths
  1. The proposed large-scale dataset combines multiple domains, and could be valuable for training generalizable 3D grounding models.
  2. The improvement in focal-length estimation and 3D grounding suggest that explicit geometric features can benefit VLM-based model for 3D grounding.
Critical weaknesses
  1. The novelty of the proposed dataset is not clear. The authors define native-camera 3D grounding as "the predicted box is expressed in the coordinate frame of the camera that captured the input" (line 032-033). However, existing datasets, such as WildDet3D, already support camera-frame 3D boxes. The Tab. 6 in the appendix distinguishes the existing datasets, but its unclear why WildDet3D is not native-frame. Its annotations already provide camera-frame 3D boxes. The proposed Omnigrounding dataset also reuses several existing 3D datasets. The benefits of the new data construction and annotation alignment are not fully demonstrated.
  2. The experiment section is weak. First, Tab. 1 mainly compares with general-purpose VLMs. The authors should also consider fine-tuning some open-source 3D grounding specialists, such as WildDet3D and DetAny3D, using the proposed data. Currently it is not clear how much of the improvements comes from the new training data and model design separately. Second, the indoor benchmarks use mean per-image [email protected] (line 358-359) instead of standard 3D detection metrics (such as mAP). The recall does not measure precision, and may not adequately reflect the grounding capability. Finally, ScanRefer seems to be the only referring expression grounding benchmark, and the performance seems very bad (5.89% [email protected] in Tab. 1). This questions the data quality of the referring expressions in Omnigrounding, and also the effectiveness of the proposed design on the referring expression grounding task.
Questions for authors
  1. Based on weakness 1, could the authors further clarify the differences between the proposed native-camera 3D grounding task and the existing data formulation (such as Wilddet3D dataset)? Why is WildDet3D marked as not supporting native-frame 3D boxes in Tab. 6?
  2. Based on weakness 2, could the authors compare against specialized 3D grounding models, such as WildDet3D and DetAny3D? Fine-tuning these models using the proposed dataset would help evaluate whether the improvements come from the dataset or the model design.
  3. Based on weakness 2, why do the authors report mean per-image [email protected] for the indoor detection benchmarks? Could the authors additionally report standard 3D detection metrics, such as mAP at different IoU thresholds?
  4. Based on weakness 2, could the authors provide further analysis of the failure cases for ScanRefer referring expression grounding? It would also be useful to evaluate on additional referring-expression grounding benchmarks.
  5. In terms of the annotation quality, how is the metric scale of game-video annotations validated? Could the authors provide some quantitative measurements?
  6. Could the authors provide a computational cost analysis, such as inference latency, and compare against VLM baselines and additional grounding specialists?
3 comments
AuthorsOpenReview ↗

Dataset Novelty, Specialists, and Detection Metrics (1/3)

Thank you for the careful review and for catching the error in Table 6. We respond to each weakness together with the questions that follow from it.

W1 & Q1: Dataset novelty and the WildDet3D entry in Table 6

"The novelty of the proposed dataset is not clear ... existing datasets, such as WildDet3D, already support camera-frame 3D boxes ... Why is WildDet3D marked as not supporting native-frame 3D boxes in Tab. 6?"

The Native-frame entry for WildDet3D in Table 6 is an error: WildDet3D provides camera-frame 3D boxes, and we will correct it.

The coordinate convention is shared with WildDet3D. OmniGrounding differs in supervision: WildDet3D-Data prompts with category names on single images, whereas OmniGrounding adds instance-level referring queries, multi-frame clips, and focal length as a generation target. It also aligns box conventions across sources, refines 1,445,349 inherited boxes, and adds open-vocabulary boxes from game videos.

Replacing the WildDet3D training data (Omni3D + WildDet3D-Data) with OmniGrounding improves [email protected] for all five models on all three benchmarks, by 4.04–9.59 points (Table R1).

Table R1. [email protected] under two training-data settings. Each cell: trained on Omni3D + WildDet3D-Data → trained on OmniGrounding. DetAny3D receives its box prompts from Grounding DINO given the text query.

ModelSUN RGB-DARKitScenesHypersim
WildDet3D34.31 → 39.6561.83 → 65.8713.45 → 19.04
DetAny3D29.74 → 34.9950.20 → 55.2610.25 → 15.53
Qwen3-VL-8B23.14 → 28.6154.75 → 64.159.14 → 14.36
Qwen3.5-9B24.69 → 30.1653.02 → 62.617.59 → 12.59
OmniGrounder36.70 → 42.0561.92 → 69.9017.19 → 22.48

W2 & Q2: Specialist comparison, data versus model

"The authors should also consider fine-tuning some open-source 3D grounding specialists, such as WildDet3D and DetAny3D, using the proposed data. Currently it is not clear how much of the improvements comes from the new training data and model design separately."

Table R1 includes the WildDet3D and DetAny3D specialists trained on OmniGrounding and separates the two factors.

  • Data. With the model fixed, OmniGrounding raises [email protected] for every model, including WildDet3D (+5.34 / +4.04 / +5.59) and DetAny3D (+5.25 / +5.06 / +5.28) on SUN RGB-D / ARKitScenes / Hypersim.

  • Model. With the data fixed to OmniGrounding, OmniGrounder exceeds its Qwen3.5-9B backbone by 11.89 / 7.29 / 9.89 and the strongest specialist, WildDet3D, by 2.40 / 4.03 / 3.44.

Q3: Detection metric

"... the indoor benchmarks use mean per-image [email protected] ... instead of standard 3D detection metrics (such as mAP). The recall does not measure precision ..."

We initially chose mean per-image [email protected] for Table 1 primarily to measure target coverage under category-conditioned queries, giving each image equal weight. However, this choice did not account for false-positive predictions. We therefore additionally report [email protected] for all models in Table R1. The IoU threshold of 0.15 is consistent with prior evaluations of VLM-based 3D box prediction, which also report average precision at this threshold [1–4]. OmniGrounder achieves the highest AP on all three benchmarks under both training-data settings.

[1] Shuai Bai et al. Qwen3-VL Technical Report. arXiv:2511.21631, 2025.

[2] Gemini Robotics Team et al. Gemini Robotics: Bringing AI into the Physical World. arXiv:2503.20020, 2025.

[3] Rui Yang et al. Visual Spatial Tuning. arXiv:2511.05491, 2025.

[4] An-Chieh Cheng et al. Grounded 3D-Aware Spatial Vision-Language Modeling. arXiv:2605.30307, 2026.

AuthorsOpenReview ↗

Referring-Expression Grounding and Metric Scale (2/3)

Q4: Referring-expression grounding

"ScanRefer seems to be the only referring expression grounding benchmark, and the performance seems very bad (5.89% [email protected] in Tab. 1) ... could the authors provide further analysis of the failure cases ... It would also be useful to evaluate on additional referring-expression grounding benchmarks."

ScanRefer failure analysis. Our ScanRefer evaluation uses RGB frames and the official referring expressions, rather than a reconstructed 3D scene. Under the Table 1 protocol, providing the ground-truth focal length raises [email protected] from 5.89 to 13.62 (+7.73 percentage points; Table R2), identifying focal-length estimation as one contributor to the low score.

Table R2. ScanRefer [email protected] under the Table 1 protocol.

ModelScanRefer [email protected]
Baseline0.79
OmniGrounder, RGB only5.89
OmniGrounder, RGB + GT focal length13.62

To analyze the remaining failures, we examined a separate post-submission evaluation using an SFT checkpoint and up to 48 input frames, covering 4,931 queries from 84 scenes whose scene IDs do not occur in our fine-tuning data. [email protected] is 5.62 with predicted focal length and 12.35 with reference focal length.

  • Non-overlapping predictions. Among the missed queries, 80.4% of the predicted boxes have zero overlap with the target, while 19.6% overlap it below the 0.25 threshold.

  • Same-category distractors. [email protected] is 16.46 on the unique subset and 6.64 on the multiple subset, which contains same-category distractors. However, 53.26% of predictions on the unique subset still have zero overlap with the target, indicating that same-category instance confusion explains only part of the failures.

These results show that focal-length error and referent ambiguity contribute to the gap, while substantial target-localization errors remain even with reference focal length. In OmniGrounding, referring queries cover six attribute groups, and each query–object correspondence is verified during data construction (Section 3.3).

Additional benchmark. We further evaluate on Multi3DRefer and report [email protected] (Table R3). OmniGrounder achieves 13.63, exceeding the baseline by 11.73 percentage points and RoboBrain 2.5-8B, the strongest of the listed VLMs, by 5.00 percentage points.

Table R3. Multi3DRefer, [email protected].

Model[email protected] (%)
Baseline1.9
Pelican-VL-7B0.29
MIMO-Embodied-8B7.5
RoboBrain 2.5-8B8.63
ACE-Brain-0-8B7.6
RynnBrain-8B1.42
OmniGrounder13.63

Q5: Validation of the metric scale of game-video annotations

"... how is the metric scale of game-video annotations validated? Could the authors provide some quantitative measurements?"

We performed a quantitative size-consistency check after Pi3X-based scale recovery, comparing the game-video box dimensions with those of the corresponding categories in real-sensor datasets: SUN RGB-D, ARKitScenes, nuScenes, KITTI, and Objectron.

  • Shared categories. Across 14 categories with at least 20 game boxes and 100 real boxes, covering 3,522 game boxes from 229 scenes, the median game-to-real size ratio is 1.00, with a 95% scene-bootstrap interval of [0.98, 1.13]. Category-level medians range from 0.63 to 1.60.

  • Cars (99 boxes). Median height, length, and width are 1.32, 4.77, and 1.84 m, compared with 1.67, 4.54, and 1.91 m in nuScenes and KITTI.

  • Standing persons (439 boxes). For person boxes whose height is at least 2.5 times the longer footprint dimension, the median height is 1.63 m, compared with 1.79 m for nuScenes and KITTI pedestrians.

These measurements provide quantitative evidence that the recovered dimensions are broadly consistent with real-world object sizes, while also documenting category-level variation. They assess metric-scale plausibility at the distribution level, rather than per-instance accuracy against engine-provided metric ground truth. Box-level quality of the same source is audited separately: on a 100-image sample, human review retains 307 of the 328 automatically accepted boxes, 295 of them without modification.

AuthorsOpenReview ↗

Inference Latency and Computational Cost (3/3)

Q6: Computational cost

"Could the authors provide a computational cost analysis, such as inference latency, and compare against VLM baselines and additional grounding specialists?"

We measured single-image inference latency on one NVIDIA H200 with batch size 1. Every model answers the same category-level 3D detection queries on 10 images, two from each of SUN RGB-D, ARKitScenes, Hypersim, KITTI, and nuScenes, with one warm-up call and three timed calls per image. The reported time is that of normally completed calls; it covers reasoning, generation, and output parsing, and excludes model loading. DetAny3D's time includes its Grounding DINO proposals.

Table R4. Single-image inference latency.

ModelLatency range (s)Mean (s)
WildDet3D0.48–1.310.74
DetAny3D0.59–1.641.06
Qwen3-VL-8B-Thinking2.62–5.694.02
Qwen3.5-9B5.98–48.3813.7
OmniGrounder3.09–7.684.23

OmniGrounder takes 4.23 s per query on average, comparable to Qwen3-VL-8B-Thinking (4.02 s) and offering a 3.2× speedup over Qwen3.5-9B (13.70 s), the backbone it builds on. The two specialists answer in about 1 s or less, since they predict boxes without autoregressive decoding, whereas the VLMs generate boxes token by token.

OmniGrounder’s reasoning trace is specific to grounding: it commits to the focal length and coarse object positions before emitting the boxes. Its outputs have a median of 108 tokens, compared with 170 for Qwen3-VL-8B-Thinking and 312 for Qwen3.5-9B. This strengthens 3D grounding while retaining the LLM-based interface for embodied reasoning and action planning; with asynchronous integration, grounding inference can remain outside the high-frequency action-control loop.

Reviewer_uPhmOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper tackles language-conditioned 3D grounding (oriented 3D box from image/video + text) in the native camera frame, arguing that apparent size entangles metric size, depth, and focal length, and that existing datasets cannot be pooled because of differing camera conventions. Contributions: (a) OmniGrounding, a corpus aligning cameras, oriented boxes, and language across indoor/driving/object-centric sources plus a new game-video source annotated semi-automatically — 4.5M boxes, 3M QA pairs, 10M focal-caption pairs; (b) OmniGrounder, a Qwen3.5-9B VLM that injects frozen VGGT-Omega reconstruction features through zero-initialized residual paths at four decoder layers, conditions box generation on supplied-or-estimated focal length, and is trained via mid-training → SFT → GRPO with geometric/format rewards. Results: gains over a same-backbone baseline on all five benchmarks, and a 23% focal-MAE reduction from geometry injection.

Critical strengths
  1. The dataset and its native-camera contract. Aligning camera intrinsics, oriented boxes, and language across heterogeneous sources — plus mining game videos for open-vocabulary boxes — directly addresses the field's data bottleneck. Releasing 4.5M boxes/3M QA with focal targets will be useful independent of the model.
  2. Well-isolated evidence that geometric priors help. The mid-training ablation (62.3→47.9 px focal MAE) and the Geo×Think factorial decomposition cleanly show that reconstruction features and the reasoning trace address complementary failure modes. This is exactly the kind of controlled evidence usually missing in VLM papers.
Critical weaknesses
  1. No training-matched external comparison. All headline gains are vs. an internal baseline. External methods in Table 1 are acknowledged as not capacity-matched, and several (Pelican-VL, MIMO-Embodied) are far below. Without a matched external baseline (e.g., Cube-LLM or LocateAnything3D fine-tuned on the same data), it is hard to attribute gains to the proposed designs vs. simply more/better data.
  2. Unquantified annotation quality for the new game-video source. The pipeline chains Pi3X scale recovery, SAM3 tracking, and VLM judging with only "light manual correction." No precision/recall audit of these boxes is reported, yet this source is presented as a key differentiator. A sampled human audit (even a few hundred boxes) would materially change confidence in the corpus.
Questions for authors
  1. What fraction of game-video candidate boxes survives each pipeline stage, and what is the measured box accuracy on a human-audited sample?
  2. ScanRefer [email protected] of 5.89% — is the bottleneck box accuracy or referent selection? An oracle-focal or oracle-2D variant would help localize the failure.
  3. In Table 5, Geo-only reduces SUN RGB-D recall (38.92 vs 42.89 Think-only). Why does the geometric prior hurt in-domain but help out-of-domain?
4 comments
AuthorsOpenReview ↗

Matched Comparisons and Annotation Quality (1/2)

Thank you for the detailed review and for recognizing the dataset's native-camera contract and the controlled evidence for geometric priors. We address the weaknesses and the questions below; W2 and Q1 concern the same annotation audit and are answered together.

W1: Training-matched external comparison

"All headline gains are vs. an internal baseline ... Without a matched external baseline (e.g., Cube-LLM or LocateAnything3D fine-tuned on the same data), it is hard to attribute gains to the proposed designs vs. simply more/better data."

Neither Cube-LLM nor LocateAnything3D has released official model weights, which precludes fine-tuning them on our data. We therefore trained two open-source 3D grounding specialists, WildDet3D and DetAny3D, together with the Qwen3-VL-8B and Qwen3.5-9B VLMs, under two training-data settings: the WildDet3D training data (Omni3D + WildDet3D-Data) and OmniGrounding. Table R1 reports [email protected] on the three indoor benchmarks. For WildDet3D under the Omni3D + WildDet3D-Data setting, we directly use its officially released weights rather than retraining it.

Table R1. [email protected] under two training-data settings. Each cell: trained on Omni3D + WildDet3D-Data → trained on OmniGrounding. DetAny3D receives its box prompts from Grounding DINO given the text query.

ModelSUN RGB-DARKitScenesHypersim
WildDet3D34.31 → 39.6561.83 → 65.8713.45 → 19.04
DetAny3D29.74 → 34.9950.20 → 55.2610.25 → 15.53
Qwen3-VL-8B23.14 → 28.6154.75 → 64.159.14 → 14.36
Qwen3.5-9B24.69 → 30.1653.02 → 62.617.59 → 12.59
OmniGrounder36.70 → 42.0561.92 → 69.9017.19 → 22.48
  • Data. With the model fixed, OmniGrounding raises [email protected] for all five models on all three benchmarks, by 4.04–9.59 points. OmniGrounder gains 5.35 / 7.98 / 5.29 points on SUN RGB-D / ARKitScenes / Hypersim, which reflects the combined value of annotation refinement and additional data.

  • Model. With the data fixed to OmniGrounding, OmniGrounder exceeds its Qwen3.5-9B backbone by 11.89 / 7.29 / 9.89 points and the strongest specialist, WildDet3D, by 2.40 / 4.03 / 3.44 points. It also achieves the highest AP on all three benchmarks under the WildDet3D training data.

These comparisons support contributions from both data construction and model design.

W2 & Q1: Annotation quality of the game-video source

"No precision/recall audit of these boxes is reported ... What fraction of game-video candidate boxes survives each pipeline stage, and what is the measured box accuracy on a human-audited sample?"

We audited the pipeline on 100 images sampled from the game videos. We recorded the boxes retained at each stage, and human review covered every box accepted by the automatic stages (Table R2).

  • Stage-wise retention. Of the 886 initial candidates, 742 remain after SAM 3 tracking and filtering, 469 after 3D lifting and box fitting, and 328 after VLM judging. The stage-wise retention rates are 83.75%, 63.21%, and 69.94%, so the automatic stages keep 37.02% of the candidates.

  • Human audit. Human review keeps 307 of the 328 automatically accepted boxes (93.60%) and removes 21. Among the 307 retained boxes, 295 require no modification and 12 (3.91%) are corrected manually, so 89.94% of the automatically accepted boxes pass the audit unchanged. The cumulative retention after review is 34.65%.

  • Metric scale. The Pi3X-based scale recovery is checked separately through a size-consistency comparison with real-sensor datasets. Over 14 shared categories (3,522 game boxes from 229 scenes), the median game-to-real size ratio is 1.00, with a 95% scene-bootstrap interval of [0.98, 1.13].

Table R2. Stage-wise retention of the game-video annotation pipeline on a 100-image sample.

Pipeline stageCandidates / boxes retainedStage retentionCumulative retention
Object discovery and candidate detection886—100.00%
SAM 3 tracking and track filtering74283.75%83.75%
3D lifting and box fitting46963.21%52.93%
VLM judging32869.94%37.02%
Human review30793.60%34.65%
AuthorsOpenReview ↗

ScanRefer Analysis, Geo–Think Ablation, and Ethics (2/2)

Q2: Bottleneck on ScanRefer

"ScanRefer [email protected] of 5.89% — is the bottleneck box accuracy or referent selection? An oracle-focal or oracle-2D variant would help localize the failure."

Oracle focal length. Under the Table 1 protocol, supplying the ground-truth focal length raises ScanRefer [email protected] from 5.89 to 13.62 (+7.73 points; Table R3). Focal-length error therefore contributes to the gap, and most queries are still missed with the correct focal length.

Table R3. ScanRefer [email protected] under the Table 1 protocol.

ModelScanRefer [email protected]
Baseline0.79
OmniGrounder, RGB only5.89
OmniGrounder, RGB + GT focal length13.62

Box accuracy versus referent selection. We further analyzed the per-query results of a separate post-submission evaluation with an SFT checkpoint and up to 48 input frames, on 4,931 queries from 84 scenes whose scene IDs do not occur in our fine-tuning data. In this setting, [email protected] is 5.62 with the predicted focal length and 12.35 with the reference focal length.

  • Coarse 3D position rather than box extent. Among the missed queries, 80.4% of the predicted boxes have zero overlap with the target, and 19.6% overlap it below the 0.25 threshold.

  • Referent selection. [email protected] is 16.46 on the unique subset and 6.64 on the multiple subset, which contains same-category distractors. On the unique subset, 53.26% of the predictions still have zero overlap, so instance confusion explains only part of the errors.

The dominant failure is therefore coarse 3D localization of the target from RGB frames, and same-category distractors reduce accuracy further.

Q3: Geo-only versus Think-only on SUN RGB-D

"In Table 5, Geo-only reduces SUN RGB-D recall (38.92 vs 42.89 Think-only). Why does the geometric prior hurt in-domain but help out-of-domain?"

Table 5 omitted the base configuration with neither the geometric prior nor the reasoning trace, so the gain of each component over this configuration was not visible. Table R4 adds it and extends the ablation to four benchmarks.

Table R4. Geo × Think ablation with the base configuration. SUN RGB-D, ARKitScenes, and Hypersim report [email protected]; WildDet3D reports AP3DdistAP_{3D}^{\mathrm{dist}}.

VariantSUN RGB-DARKitScenesHypersimWildDet3D
Base (no Geo, no Think)30.1650.2913.2119.34
Think-only42.8958.317.2120.16
Geo-only38.9260.5816.8430.68
Geo+Think44.6264.6519.7731.48

Both Geo-only and Think-only improve over the base configuration on all four benchmarks. On SUN RGB-D, the geometric prior improves recall in both arms of the ablation: from 30.16 to 38.92 without the reasoning trace, and from 42.89 to 44.62 with it. Geo-only is below Think-only on this benchmark because the trace alone gives a larger gain (30.16 to 42.89). The relative strength of the two components varies across benchmarks: Think-only is higher on SUN RGB-D and Hypersim, and Geo-only is higher on ARKitScenes and WildDet3D. Geo+Think is the best configuration on all four benchmarks, which supports the complementarity of the two components. We will add the base configuration to Table 5 in the revision.

Ethics flag

We note that the reviewer selected the ethics flag and we appreciate their attention to the ethical aspects of our data. We would like to provide further context: the original data used in this work come from publicly available datasets, including the game videos from OmniWorld-Game (Zhou et al., 2025), and are used in accordance with their respective licenses and terms of use. As described in our Ethics Statement, this study does not involve human subjects or personally identifiable information.

Thank you for the detailed response. The new comparisons and annotation audit address many of my concerns, and the additional ScanRefer and Geo–Think analyses are helpful. I still see value in the dataset and native-camera formulation, and I lean weakly positive.

Thank you for taking the time to read our responses and follow up. Your questions helped us improve the comparisons, document the annotation quality, and clarify the ScanRefer and Geo–Think results. We are glad you found these additions helpful, and we sincerely appreciate your continued recognition of the dataset and native-camera formulation.

Reviewer_9FhLOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes an active-camera 3D grounding dataset that aligns cameras, boxes, and language across indoor, driving, object-centric, and game-video sources at scale. It also uses geometric features extracted from a frozen VGGT-Omega encoder and a focal-length token-reconstruction design to train the model for native-camera 3D grounding.

Critical strengths
  1. This paper integrates visual observations, camera parameters, oriented 3D boxes, and language annotations into a unified dataset.

  2. This paper has fluent and clear writing and layout, easy to read and follow.

Critical weaknesses

1. The method uses geometric features extracted from a frozen VGGT-Omega encoder. Have the authors evaluated alternative feature extraction backbones? An ablation study across different backbones would help assess the method’s dependence on the chosen encoder and provide stronger evidence of its generality.

2. I appreciate the complementary nature of indoor and outdoor source data and find this strategy reasonable. However, the outdoor data appear to come exclusively from driving scenarios, which raises concerns about limited scene diversity and an overrepresentation of roads and vehicles. Are the game videos also primarily driving-related, as the examples in Figure 1 suggest?

3. Also, A more detailed analysis of the dataset composition would be helpful, including the proportion contributed by each data source and the categories and relative frequencies of scenarios covered by the game videos.

Questions for authors

Attached in the weakness section.

2 comments
AuthorsOpenReview ↗

Geometric Encoders, Outdoor Diversity, and Dataset Composition

Thank you for the positive assessment of the unified dataset and the presentation.

W1: Alternative geometric encoders

"Have the authors evaluated alternative feature extraction backbones? An ablation study across different backbones would help assess the method's dependence on the chosen encoder ..."

We added an ablation that replaces VGGT-Omega with two alternative geometric encoders, VGGT and MoGe-2, and trains each variant under the setting of the final row of Table 1, which includes WildDet3D. VGGT uses the same feature-extraction configuration as VGGT-Omega. For MoGe-2, we extract features from layers [5, 11, 17, 23] to accommodate its architecture and feed them into the geometry fusion module.

Table R1. Geometric-encoder ablation ([email protected]). The three encoder rows share one training setting, which includes WildDet3D.

Geometric encoderSUN RGB-DARKitScenesHypersim
Baseline42.8958.317.21
VGGT44.6866.3221.98
MoGe-245.7468.1920.87
VGGT-Omega (default)48.9568.3222.74

The geometry fusion module operates with all three encoders, so the design is not specific to VGGT-Omega. Each variant exceeds the baseline on all three benchmarks; the data-matched comparison for VGGT-Omega is in Table 1 (42.89 → 44.62, 58.30 → 64.65, and 17.21 → 19.77 without WildDet3D training). Among the three encoders, VGGT-Omega gives the highest recall on every benchmark.

W2: Outdoor scene diversity and the content of the game videos

"... the outdoor data appear to come exclusively from driving scenarios ... Are the game videos also primarily driving-related, as the examples in Figure 1 suggest?"

Outdoor supervision in OmniGrounding extends beyond driving. The WildDet3D sources contribute outdoor scenes from urban settings, such as sports activities and markets, and from natural environments, such as parks, mountains, and beaches. Under a preliminary scene categorization (Table R2), road-traffic and street scenes account for 37.64% of the outdoor QA pairs, and the remaining 62.36% come from other outdoor scenes.

The game videos are likewise not primarily driving-related. Road-traffic and street scenes account for 36.65% of the outdoor QA pairs from OmniWorld, and the remaining 63.35% belong to other scene types. Figure 1 shows a single annotation example, selected to illustrate the pipeline.

Table R2. Outdoor QA pairs by source under a preliminary scene categorization.

SourceOutdoor QA pairsRoad-traffic / street-scene QA pairsShare
WildDet3D sources1,211,057429,56435.47%
nuScenes38,15938,159100.00%
KITTI6,2476,247100.00%
OmniWorld (game videos)12,6884,65036.65%
Other sources combined4,4143858.72%
Total1,272,565479,00537.64%

W3: Dataset composition

"A more detailed analysis of the dataset composition would be helpful, including the proportion contributed by each data source and the categories and relative frequencies of scenarios covered by the game videos."

OmniGrounding contains 3,645,538 QA pairs, of which 3,609,018 are used for training and 36,520 for validation. Table R3 reports the contribution of each source and the number of refined boxes incorporated into the corpus. WildDet3D contributes the largest share of QA pairs because it aggregates in-the-wild data from multiple sources and domains. Annotation refinement covers 1,445,349 boxes, of which 959,797 come from WildDet3D and 484,946 from HoliSpatial / ScanNet++ / ScanNet v2. For the game videos, Table R2 reports the share of road-traffic and street scenes among their outdoor QA pairs.

Table R3. Composition of OmniGrounding by source.

SourceTraining QATotal QARefined boxes incorporated
WildDet3D2,550,4712,584,942959,797
nuScenes38,61038,941—
ARKitScenes36,09336,368—
Hypersim24,73325,303—
OmniWorld (video)446,650446,807—
Objectron9,95610,055560
KITTI5,6926,25146
SUN RGB-D4,0024,042—
HoliSpatial / ScanNet++ / ScanNet v2 (video)492,811492,829484,946
Total3,609,0183,645,5381,445,349

We thank you again for the constructive suggestions, which helped us strengthen the geometric-encoder evaluation and clarify the diversity and composition of OmniGrounding.

Dear Reviewer 9FhL,

Thank you for your suggestions on the geometric encoder and dataset composition. In our response, we added comparisons with VGGT and MoGe-2. We also reported each data source’s contribution and the share of road and street scenes among outdoor QA pairs, including those from the OmniWorld game videos.

Beyond expanding source coverage, OmniGrounding incorporates 1,445,349 refined 3D boxes and provides grounding-oriented reasoning traces in its QA pairs. These traces train the model to describe the target before predicting its 3D box.

The revised manuscript is now available on OpenReview. We would appreciate a brief note on whether the added encoder comparisons and dataset breakdown address your questions. If any points remain open, we would be happy to clarify them.

Best regards,

The Authors of Submission #10469

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.