acceptodds
Under review as a conference paper at ICLR 2027

OMG: OPEN-MODALITY TRAINING-FREE 3D VISUAL GROUNDING

Abstract

3D visual grounding requires reasoning over spatial relationships and object at- tributes under diverse observation conditions. We propose a modality-unified, training-free framework that supports point clouds, posed RGB-D observations, and unposed multi-view RGB images through a shared grounding procedure. Modality-specific perception front-ends convert observations into semantic 3D instances, enabling query interpretation and reasoning through a common object- level interface. At its core, query-conditioned spatial abstraction preserves rele- vant target–reference configurations and scene context, while local observations provide appearance and fine geometric details. Candidate-adaptive verification jointly evaluates these complementary sources of evidence, introducing an addi- tional spatial filtering step only when the candidate count exceeds a fixed budget. Excluding scene preprocessing, grounding requires at most three model calls per query without task-specific parameter updates. Under point-cloud-only input, our method achieves 51.9% [email protected] and 46.7% [email protected] on ScanRefer, exceed- ing the reported SeeGround results by 7.8 and 7.3 percentage points, and reaches 50.18% accuracy on Nr3D. The same grounding procedure and hyperparameters achieve 48.1% and 34.4% [email protected] on ScanRefer with RGB-D and unposed RGB inputs, respectively. Controlled ablations support the effectiveness of combining compact spatial abstraction with local evidence for grounding across observation modalities.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D visual grounding, 3D scene understanding, vision-language models, multimodal reasoning, referring expression comprehension, open-vocabulary 3D perception

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_fjXjOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

OMG is a training-free 3D visual grounding framework supporting point clouds, posed RGB-D, and unposed multi-view RGB. Modality-specific perception produces a common interface of semantic 3D instances and local evidence. Role-aware category mapping selects targets and references, which are rendered as a cuboid spatial abstraction. A candidate-count-based procedure either directly verifies a small candidate set or first selects the top four spatial candidates. Experiments report ScanRefer localization results across the three inputs and Nr3D candidate-grounding results, with point-cloud component ablations and an inference-cost comparison.

Critical strengths
  1. The separation between a common spatial interface and modality-specific local evidence is coherent and practically useful. The role-aware mapping and bounded verification procedure are specified concretely, making the framework more informative than simply invoking a generic VLM over a scene.

  2. The point-cloud controls provide meaningful evidence. Table 3 supports substantial contributions from mapping and verification, while Tables 5–6 show that the simplified spatial representation improves over bbox text and largely preserves rendering-based accuracy with matched local evidence. Appendix A also compares against SeqVLM with the same Qwen model and reports fewer calls and lower peak token context alongside higher accuracy.

Critical weaknesses
  1. The central cross-modality gains are not sufficiently separated from perception and backbone choices. In the RGB-only block of Table 1, OMG uses SegVGGT while the compared systems use DUSt3R, so the 12.8/21.2-point gain cannot be attributed to the shared reasoning design. The controlled component experiments are point-cloud-only. I recognize the larger mapping/verification gains and the 2.42/2.01-point pre-verification benefit of abstraction; the framework's contribution is not just its 0.64/0.60-point final marginal gain. However, Table 6 shows near-equivalent rather than superior accuracy to rendering, and neither cross-modal robustness nor abstraction-specific cost benefits are isolated. This leaves the distinctive methodological value incompletely established.

  2. The unposed-RGB evaluation requires a clearer coordinate and scale protocol. Equation (1) outputs a scene-frame AABB, and ScanRefer computes 3D IoU against ScanNet-frame metric boxes, but the manuscript does not explain how the reconstructed geometry is transformed into that frame, including scale and vertical orientation. B.1 specifies posed RGB-D projection, while B.2 assumes a vertical axis and a fixed camera offset. This matters because SegVGGT's published limitations explicitly state that it does not recover absolute metric scale. An existing alignment procedure may resolve the issue, but its inputs and role in reasoning versus evaluation need to be disclosed before the RGB-only results can be fully interpreted.

Questions for authors
  1. Under fixed perception outputs, local evidence, and the same VLM, how much does OMG improve over an existing strong grounding procedure or the bbox-text/verification control for RGB-D and unposed RGB? A bounded comparison with candidate recall, filtering, and final accuracy would help isolate the shared reasoning contribution. If compactness is a central advantage, please also quantify the matched abstraction-versus-rendering cost.

  2. Please specify the complete unposed-RGB preprocessing and evaluation pipeline: image sampling, checkpoint/category vocabulary, scale/orientation/translation alignment, and box fitting. What additional information is used, and is it available to reasoning or only to evaluation? Please state whether the compared RGB baselines use the same alignment convention.

3 comments

Thank you for your constructive feedback and for recognizing the framework’s design and component analyses.

1. Cross-Modal Gains and the Independent Contribution of Shared Reasoning

  1. We agree that comparisons across perception frontends cannot isolate grounding contributions. We therefore evaluate OMG and Z3D on the same 250-query Nr3D subset with identical GT candidate boxes and Qwen3.8-27B-AWQ weights, testing Z3D in both Instruct and Thinking modes.
  2. OMG achieves accuracy comparable to Z3D Thinking (58.80% vs. 58.40%) and exceeds Z3D Instruct by 8.8 percentage points. It reduces calls by 90.3% and 88.3%, respectively, and lowers time per query to 49.65 s,demonstrating competitive grounding accuracy at lower inference cost under controlled candidate inputs and model weights.
MethodModeAvg. Calls ↓Peak Input Tokens ↓Peak Input + Output Tokens ↓Total Tokens ↓Time/Query ↓Overall ↑
Z3DInstruct17.30019,63920,87133,543,222332.00 s50.00%
Z3DThinking20.90420,77723,19140,709,2121,822.00 s58.40%
OMGInstruct2.0321,5883,636687,49549.65 s58.80%

2. Accuracy and Cost Trade-offs Between Spatial Abstraction and Point-Cloud Rendering

Constructing instance-level 3D bounding boxes requires far less geometric information than reconstructing a scene point cloud. Using only these boxes for rendering, spatial abstraction achieves performance close to that of point-cloud rendering and outperforms it on some metrics. When these boxes are available, it also avoids the additional cost of reconstructing scene point clouds for certain input modalities.

  1. We measure representation rendering time, excluding perception, reconstruction, and VLM inference.GPU point-cloud rendering is faster for smaller point clouds. However, point-cloud rendering time increases substantially with the number of scene points, whereas spatial abstraction rendering remains within 0.33–0.47 s.Spatial abstraction therefore incurs lower rendering costs for the evaluated scenes containing millions of points.
PointsCPU Point-Cloud Rendering (s)GPU Point-Cloud Rendering (s)Spatial Abstraction (s)
230K–580K0.47–1.200.14–0.200.45–0.47
3.05M5.560.660.45
5.28M10.871.150.45
9.14M19.551.660.33
  1. When point clouds are unavailable, point-cloud rendering requires acquiring or reconstructing scene geometry. ScanNet reports a median reconstruction time of 11.3 minutes per scene [1]. This suggests that scene reconstruction may incur non-negligible preprocessing costs.

3. Unposed RGB protocol and GT information usage. Our original pipeline followed the instance segmentation evaluation setup in SegVGGT [2], mapping predicted masks onto the official reconstructed geometry to construct instance proposals. We have revised the pipeline to construct instance geometry directly from SegVGGT-predicted depths, camera parameters, and instance masks. We use the GT depth and pose of a single reference frame, together with the official scene axis-alignment matrix, to transform the predicted geometry into the metric evaluation coordinate system. Z3D’s released unposed-RGB preprocessing uses the same types of alignment information. The protocol and implementation details have been added to Section 4.1 and Appendix B.1, with all revisions highlighted in blue. The revised Overall [email protected]/0.50 is 35.64/18.24.

[1] Dai et al. ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes. CVPR, 2017.

[2] Jinyuan Qu, Hongyang Li, and Lei Zhang. Segvggt: Joint 3d reconstruction and instance segmentation from multi-view images.

Reviewer_fjXjOpenReview ↗

Response to Authors

Thank the authors for the detailed response and additional experiments. Especially, the author revised the unposed-RGB experiment and adding the controlled Nr3D comparison.

However, my concerns remain since there are still ambiguities: e.g., is the revised unposed-RGB comparison matched in calibration? What does the matched comparison establish with predicted proposals? I encourage the authors to further clarify these ambiguities.

Thank you for your constructive feedback.

In the revised comparison with unposed RGB inputs, both methods use the same camera calibration settings.

OMG builds a spatial abstraction graph from object proposals and uses it to reason about spatial relationships, whereas Z3D reasons directly from 2D images. These approaches involve different trade-offs.

  1. The spatial abstraction graph enables OMG to handle multiple input modalities, including point-cloud-only inputs, which Z3D currently does not support. Graph-based reasoning also contributes to OMG’s tens-fold inference speedup in the reported experimental settings.

  2. Errors in the proposals can propagate to downstream reasoning. Table 1in paper compares OMG and Z3D in the setting where OMG uses Mask3D proposals. In our previous response, we also reported results using ground-truth proposals. In the comparison using predicted proposals, Z3D achieves somewhat higher accuracy.

Reviewer_opMTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a training-free framework for 3D visual grounding, which represents relevant objects using simplified 3D boxes and uses local visual or geometric information to distinguish ambiguous candidates. It evaluates OMG on ScanRefer and Nr3D and report competitive results across the supported modalities. Ablation studies further examine the contributions of category mapping, spatial abstraction and verification.

Critical strengths
  1. Different sensing setups provide different types of observations. The proposed instance-level interface makes this possible without requiring separate grounding models for point clouds, RGB-D, and RGB inputs. This is a reasonable and potentially useful design choice.
  2. The results on ScanRefer and Nr3D are encouraging. The paper examines category mapping, spatial representation, and verification separately rather than reporting only the final performance.
Critical weaknesses
  1. The cross-modality comparisons do not fully isolate the contribution of the grounding framework. In the unposed RGB setting, OMG uses SegVGGT to obtain 3D object proposals, while the compared methods use different reconstruction and perception pipelines. It is difficult to determine how much of the performance improvement comes from the proposed grounding procedure and how much comes from better upstream 3D reconstruction and instance proposals.
  2. The individual contribution of spatial abstraction appears relatively modest. Although query-conditioned spatial abstraction is presented as a central component, its effect on the final grounding accuracy is relatively small. In Table 3, removing it reduces [email protected] from 51.27% to 50.63%. Table 6 also shows that point-cloud rendering achieves slightly higher final accuracy than spatial abstraction when both use the same local evidence. The improvements seem to come more substantially from category mapping and joint verification than from the spatial abstraction itself.
Questions for authors
  1. How is the scene mesh obtained? Does the method use any information beyond the input point cloud? Please clarify the exact input assumptions.
  2. How to provide a comparison using the same perception front-end or identical 3D object proposals? This would help distinguish the improvements from SegVGGT-based scene reconstruction from those of the proposed grounding framework.
  3. How to provide more direct evidence of its advantages over point-cloud rendering, such as end-to-end latency, preprocessing cost, or robustness across input modalities? The current inference-cost analysis mainly reports model calls and context size.
4 comments

Thank you for your insightful comments. We provide additional controlled experiments and cost measurements below.

1. Separating perception from grounding.

  1. We agree that comparisons across perception frontends cannot isolate grounding contributions. We therefore evaluate OMG and Z3D on the same 250-query Nr3D subset with identical GT candidate boxes and Qwen3.8-27B-AWQ weights, testing Z3D in both Instruct and Thinking modes.
  2. OMG achieves accuracy comparable to Z3D Thinking (58.80% vs. 58.40%) and exceeds Z3D Instruct by 8.8 percentage points. It reduces calls by 90.3% and 88.3%, respectively, and lowers time per query to 49.65 s,demonstrating competitive grounding accuracy at lower inference cost under controlled candidate inputs and model weights.
MethodModeAvg. Calls↓Peak Input Tokens↓Peak Input + Output Tokens↓Total Tokens↓Time/Query↓Overall↑
Z3DInstruct17.30019,63920,87133,543,222332.00 s50.00%
Z3DThinking20.90420,77723,19140,709,2121,822.00 s58.40%
OMGInstruct2.0321,5883,636687,49549.65 s58.80%

2. Gains from spatial abstraction.

  1. Spatial abstraction clearly improves candidate selection, while subsequent modules compensate for performance differences between representations to some extent. Table 4 in the main paper isolates this contribution on subsets of ScanRefer and Nr3D containing more than four candidates, with the candidate pool, VLM, and Top-K fixed. Compared with textual box coordinates, spatial abstraction improves [email protected] and [email protected] by 3.02 and 3.52 percentage points on ScanRefer, and by 4.60 and 2.69 percentage points on Nr3D, respectively.
  2. It also performs comparably to point-cloud rendering, with gains on some metrics, while requiring only instance-level 3D boxes. Thus, when these boxes are available, RGB inputs require no full-scene reconstruction solely for first-stage rendering.

3. Scene meshes and input assumptions. Our original meshes are ScanNet reconstructions[1]. To test dependence on these specific meshes, we replace them with meshes reconstructed from input point clouds using BPA on the same ScanRefer subset, keeping all other settings fixed.The results are similar, indicating that indicating that meshes reconstructed from point clouds are also sufficiently effective. We will clarify the mesh sources and input requirements.

Stage-Two Local View[email protected][email protected]
Original mesh + CoV53.2%47.2%
BPA mesh from point cloud + CoV53.6%46.8%

4. Rendering cost, latency, and cross-modal robustness.

  1. We measure representation rendering time, excluding perception, reconstruction, and VLM inference.GPU point-cloud rendering is faster for smaller point clouds. However, point-cloud rendering time increases substantially with the number of scene points, whereas spatial abstraction rendering remains within 0.33–0.47 s.Spatial abstraction therefore incurs lower rendering costs for the evaluated scenes containing millions of points.
PointsCPU Point-Cloud Rendering (s)GPU Point-Cloud Rendering (s)Spatial Abstraction (s)
230K–580K0.47–1.200.14–0.200.45–0.47
3.05M5.560.660.45
5.28M10.871.150.45
9.14M19.551.660.33
  1. When point clouds are unavailable, point-cloud rendering requires acquiring or reconstructing scene geometry. ScanNet reports a median reconstruction time of 11.3 minutes per scene [1]. This suggests that scene reconstruction may incur non-negligible preprocessing costs.

[1] Dai et al. ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes. CVPR, 2017.

Reviewer_opMTOpenReview ↗

Reply to the paper

Thank you for the detailed rebuttal and additional experiments. I appreciate the authors' efforts to address my concerns. My remaining concern is primarily about the contribution of spatial abstraction. I encourage the authors to incorporate these additional experiments and clarify the specific advantages and limitations of spatial abstraction relative to point-cloud rendering.

Thank you for your positive response.

Specific advantages.

  1. We introduce a novel instance-level representation that is applicable even when the input modalities do not include point clouds.

  2. This representation also offers efficiency advantages over directly using point clouds. In our previous response, we reported a quantitative comparison of rendering costs, as well as the time cost of point-cloud reconstruction.

Specific limitations. Spatial abstraction preserves core spatial information while reducing potentially distracting visual details. However, this simplification inevitably discards fine-grained geometry and appearance cues that may help answer certain grounding queries. It may therefore limit the ability of more capable models to fully exploit rich scene information.

Please let us know if you have any further questions or concerns.

Thank you for the additional clarification. The experiments demonstrate rendering efficiency advantages for large point clouds. However, I remain somewhat unconvinced about the significance of spatial abstraction as an individual technical contribution within the complete framework. The new rendering benchmarks exclude upstream perception and VLM inference, so they mainly establish representation-level efficiency rather than the end-to-end benefit attributable to spatial abstraction.

Reviewer_G8LXOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces OMG (Open-Modality Training-Free 3D Visual Grounding), a unified, training-free framework designed to ground natural-language queries into 3D bounding boxes across heterogeneous sensing inputs, namely colored point clouds, posed RGB-D frames, and unposed multi-view RGB images. Addressing the limitation that existing 3DVG methods are tightly coupled to a single specific input modality, the authors propose to decouple modality-dependent scene perception from downstream language-guided spatial reasoning. OMG is evaluated on two standard benchmarks: ScanRefer and Nr3D. The experimental results show that OMG establishes new state-of-the-art zero-shot/training-free results on ScanRefer under both point-cloud and unposed RGB settings.

Critical strengths
  1. Existing 3DVG methods generally suffer from a "modality-binding" issue, yet real-world embodied AI systems utilize a highly diverse range of sensors. This work astutely identifies this pain point and decouples "perception" from "reasoning," proposing a unified, general-purpose framework capable of handling point clouds, RGB-D data, and unposed RGB images—a design with significant practical value for real-world deployment.

  2. The proposed OMG method was evaluated across three input modalities on two core benchmarks: ScanRefer and Nr3D. In the pure point-cloud setting, it establishes a new state-of-the-art (SOTA) for zero-shot performance (achieving improvements of 7.2% and 6.8% over SeeGround on the ScanRefer dataset). Furthermore, it significantly outperforms baselines when using the highly challenging "unposed RGB image" input. At the same time, the method maintains low computational hardware requirements and efficient processing times.

Critical weaknesses
  1. The technical innovation is somewhat limited. Although the paper emphasizes the "unification of inputs from different modalities," the core approach essentially relies on various candidate detectors (such as Mask3D and SegVGGT) to convert inputs into 3D bounding boxes, which are then passed to a VLM for subsequent processing. This is a common engineering-oriented strategy; the challenge of modal heterogeneity is primarily addressed by existing off-the-shelf perception models, rendering the contribution regarding "open-modality" capabilities at the algorithmic level somewhat weak.

  2. OMG shows a substantial gap between view-dependent and view-independent queries on Nr3D (41.7% vs. 54.4%). Since the abstraction camera is not aligned with the viewpoint implied by the query (Appendix B.2), it remains unclear how the framework resolves query-specific reference frames. Additional analysis or a viewpoint-handling ablation would help explain this limitation.

Questions for authors

The framework heavily assumes that the perception front-ends (Mask3D / SegVGGT) successfully capture the target bounding box. What is the upper-bound performance of OMG if provided with Ground-Truth (Oracle) 3D proposals?

1 comment

We thank the reviewer for the insightful comments and for recognizing the practical value of OMG.

1. Technical novelty and reliance on off-the-shelf perception frontends. OMG’s contribution lies in organizing instance information around the query and designing a unified spatial reasoning pipeline.

  1. Our spatial representation can be constructed from instance-level information provided by different modalities without relying on point-cloud rendering, thereby providing a unified interface for open-modality 3D grounding.This is also the main reason why our method can address the “modality-binding” issue.
  2. To the best of our knowledge, this is also the first work on open-modality 3D grounding, aiming to introduce new research directions and application scenarios to the community.

2. Performance gap between view-dependent and view-independent queries.

  1. We do not follow prior work in first inferring the view described in the query and then rendering an image from that view, as discrepancies between the rendered view and the actual query-described view may introduce additional errors. Instead, we use a spatial abstraction image to objectively represent the scene’s original spatial layout. We explicitly distinguish the spatial abstraction view from the query-described view in the prompt, guiding the model to infer the latter from the original layout. For view-dependent queries, the model can also use the original frames as supplementary visual evidence.

  2. The spatial abstraction image directly presents the overall scene layout and is therefore better suited to view-independent queries. We hypothesize that stronger models may have greater view inference capabilities and conduct the following experiments on Nr3D to examine this hypothesis. The results show that performance on view-dependent queries is affected by model capability: as model capability improves, the performance gap between view-dependent and view-independent queries narrows.

Model / Verification ConfigurationOverallView-DependentView-IndependentGap (Percentage Points)
Qwen3-VL-32B50.18%41.69%54.38%12.69
Qwen3.8-27B55.43%48.95%58.64%9.69
JEV[1]54.54%49.48%57.04%7.56

After replacing Qwen3-VL-32B with Qwen3.8-27B, accuracy on viewpoint-related questions increased from 41.69% to 48.95%, while the accuracy gap between viewpoint-related and viewpoint-independent questions narrowed from 12.69 to 9.69 percentage points. With the further introduction of the Jev decision model based on Qwen3.8-27B, accuracy on viewpoint-related questions increased to 49.48%, and the gap narrowed to 7.56 percentage points.

3. Performance with Ground-Truth Proposals. To analyze the effect of instance information quality on OMG, we conducted an additional experiment on ScanRefer using ground-truth (GT) instance bounding boxes and categories. With Qwen3-VL-32B, replacing Mask3D-predicted instances with GT bounding boxes and categories increased OMG’s [email protected] from 53.1% to 66.43%, a gain of 13.33 percentage points. Compared with the original SeeGround experiment using Mask3D and Qwen2-VL-72B, OMG achieved higher accuracy with both Mask3D (53.1% vs. 44.1%) and GT instances (66.43% vs. 44.1%). Furthermore, under the same Mask3D and Qwen3-VL-32B setting, OMG outperformed SeeGround by 4.8 percentage points, demonstrating the additional benefit of its spatial representation and reasoning pipeline.

MethodInstance SourceVLMOverall [email protected]
SeeGroundMask3DQwen2-VL-72B44.1%
SeeGroundGTQwen2-VL-72B59.5%
SeeGroundMask3DQwen3-VL-32B48.3%
OMGMask3DQwen3-VL-32B53.1%
OMGGTQwen3-VL-32B66.43%

[1] Almeida, D. (2026). Introducing System One Models & Jev. TypeSafe AI.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.