- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
OMG is a training-free 3D visual grounding framework supporting point clouds, posed RGB-D, and unposed multi-view RGB. Modality-specific perception produces a common interface of semantic 3D instances and local evidence. Role-aware category mapping selects targets and references, which are rendered as a cuboid spatial abstraction. A candidate-count-based procedure either directly verifies a small candidate set or first selects the top four spatial candidates. Experiments report ScanRefer localization results across the three inputs and Nr3D candidate-grounding results, with point-cloud component ablations and an inference-cost comparison.
Critical strengths
-
The separation between a common spatial interface and modality-specific local evidence is coherent and practically useful. The role-aware mapping and bounded verification procedure are specified concretely, making the framework more informative than simply invoking a generic VLM over a scene.
-
The point-cloud controls provide meaningful evidence. Table 3 supports substantial contributions from mapping and verification, while Tables 5–6 show that the simplified spatial representation improves over bbox text and largely preserves rendering-based accuracy with matched local evidence. Appendix A also compares against SeqVLM with the same Qwen model and reports fewer calls and lower peak token context alongside higher accuracy.
Critical weaknesses
-
The central cross-modality gains are not sufficiently separated from perception and backbone choices. In the RGB-only block of Table 1, OMG uses SegVGGT while the compared systems use DUSt3R, so the 12.8/21.2-point gain cannot be attributed to the shared reasoning design. The controlled component experiments are point-cloud-only. I recognize the larger mapping/verification gains and the 2.42/2.01-point pre-verification benefit of abstraction; the framework's contribution is not just its 0.64/0.60-point final marginal gain. However, Table 6 shows near-equivalent rather than superior accuracy to rendering, and neither cross-modal robustness nor abstraction-specific cost benefits are isolated. This leaves the distinctive methodological value incompletely established.
-
The unposed-RGB evaluation requires a clearer coordinate and scale protocol. Equation (1) outputs a scene-frame AABB, and ScanRefer computes 3D IoU against ScanNet-frame metric boxes, but the manuscript does not explain how the reconstructed geometry is transformed into that frame, including scale and vertical orientation. B.1 specifies posed RGB-D projection, while B.2 assumes a vertical axis and a fixed camera offset. This matters because SegVGGT's published limitations explicitly state that it does not recover absolute metric scale. An existing alignment procedure may resolve the issue, but its inputs and role in reasoning versus evaluation need to be disclosed before the RGB-only results can be fully interpreted.
Questions for authors
-
Under fixed perception outputs, local evidence, and the same VLM, how much does OMG improve over an existing strong grounding procedure or the bbox-text/verification control for RGB-D and unposed RGB? A bounded comparison with candidate recall, filtering, and final accuracy would help isolate the shared reasoning contribution. If compactness is a central advantage, please also quantify the matched abstraction-versus-rendering cost.
-
Please specify the complete unposed-RGB preprocessing and evaluation pipeline: image sampling, checkpoint/category vocabulary, scale/orientation/translation alignment, and box fitting. What additional information is used, and is it available to reasoning or only to evaluation? Please state whether the compared RGB baselines use the same alignment convention.
3 comments
Response to Authors
Thank the authors for the detailed response and additional experiments. Especially, the author revised the unposed-RGB experiment and adding the controlled Nr3D comparison.
However, my concerns remain since there are still ambiguities: e.g., is the revised unposed-RGB comparison matched in calibration? What does the matched comparison establish with predicted proposals? I encourage the authors to further clarify these ambiguities.
Thank you for your constructive feedback.
In the revised comparison with unposed RGB inputs, both methods use the same camera calibration settings.
OMG builds a spatial abstraction graph from object proposals and uses it to reason about spatial relationships, whereas Z3D reasons directly from 2D images. These approaches involve different trade-offs.
-
The spatial abstraction graph enables OMG to handle multiple input modalities, including point-cloud-only inputs, which Z3D currently does not support. Graph-based reasoning also contributes to OMG’s tens-fold inference speedup in the reported experimental settings.
-
Errors in the proposals can propagate to downstream reasoning. Table 1in paper compares OMG and Z3D in the setting where OMG uses Mask3D proposals. In our previous response, we also reported results using ground-truth proposals. In the comparison using predicted proposals, Z3D achieves somewhat higher accuracy.
Thank you for your constructive feedback and for recognizing the framework’s design and component analyses.
1. Cross-Modal Gains and the Independent Contribution of Shared Reasoning
2. Accuracy and Cost Trade-offs Between Spatial Abstraction and Point-Cloud Rendering
Constructing instance-level 3D bounding boxes requires far less geometric information than reconstructing a scene point cloud. Using only these boxes for rendering, spatial abstraction achieves performance close to that of point-cloud rendering and outperforms it on some metrics. When these boxes are available, it also avoids the additional cost of reconstructing scene point clouds for certain input modalities.
3. Unposed RGB protocol and GT information usage. Our original pipeline followed the instance segmentation evaluation setup in SegVGGT [2], mapping predicted masks onto the official reconstructed geometry to construct instance proposals. We have revised the pipeline to construct instance geometry directly from SegVGGT-predicted depths, camera parameters, and instance masks. We use the GT depth and pose of a single reference frame, together with the official scene axis-alignment matrix, to transform the predicted geometry into the metric evaluation coordinate system. Z3D’s released unposed-RGB preprocessing uses the same types of alignment information. The protocol and implementation details have been added to Section 4.1 and Appendix B.1, with all revisions highlighted in blue. The revised Overall [email protected]/0.50 is 35.64/18.24.
[1] Dai et al. ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes. CVPR, 2017.
[2] Jinyuan Qu, Hongyang Li, and Lei Zhang. Segvggt: Joint 3d reconstruction and instance segmentation from multi-view images.