- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies referential monocular 3D object localization: predicting a metric camera-frame 3D bounding box for an object specified by a natural-language query in a single RGB image. The proposed framework combines frozen VLM-based 2D grounding, instance segmentation, monocular metric geometry estimation, gravity estimation, point-cloud filtering, and gravity-constrained box fitting. It also introduces Ref3DOL, constructed from Omni3D-unified SUN RGB-D and Hypersim annotations, with 33,915 category-level and 41,875 instance-level queries. The best configuration outperforms the evaluated direct 3D-box prediction baselines, while ablations and oracle experiments examine several sources of error.
Critical strengths
-
A practical framework with measurable gains over the evaluated direct predictors. The method requires no task-specific parameter updates and supports several grounding backbones. On the instance-level track, its best configuration improves [email protected] over the strongest evaluated direct baseline by 5.15 points on SUN RGB-D and 9.33 points on Hypersim. These results support the usefulness of composing pretrained semantic and geometric capabilities as an alternative inference pipeline.
-
Useful component and oracle analyses. The grounding and segmentation ablations show the importance of resolving the referred instance and isolating its pixels. On SUN RGB-D with ground-truth intrinsics, replacing predicted depth with ground-truth depth raises [email protected] from 35.25 to 60.67, compared with 43.01 when replacing only the 2D box. These experiments provide actionable evidence that metric depth is a major bottleneck in the evaluated system.
Critical weaknesses
-
The methodological contribution relative to closely related geometric pipelines is insufficiently established. OVMono3D-GEO already combines pretrained 2D detection, segmentation, metric depth, geometric unprojection, and point-cloud processing without task-specific training. The pseudo-box generation stage of OVM3D-Det also includes geometry-based box construction and ground-aligned orientation estimation, although its complete framework subsequently trains a detector. These related pipelines are not discussed. Their open-vocabulary detection setting differs from referential localization, but their geometric components provide relevant points of comparison. The current comparisons against direct VLM predictions do not isolate the contribution of the proposed reconstruction and fitting design from the benefit of using dedicated pretrained geometry models. An adapted geometric baseline with matched grounding, masks, depth, and intrinsics would help establish the incremental contribution.
-
The generated instance-level queries need stronger independent validation. Attribute extraction and the principal automatic verification both use Qwen3-VL-32B. The reported manual inspection of incorrect selections is useful, but it does not independently audit a random sample of all queries or exclude errors shared by the generation and verification stages. Appendix A also maps camera-frame X/Y coordinate order to image-relative ordering expressions. Under perspective projection, image position depends on X/Z and Y/Z, so the stated ordering rules do not guarantee consistency with image-space order when depths differ. This identifies a potential failure mode rather than demonstrating its prevalence in Ref3DOL. An independent query audit and a projection-consistency analysis are needed to assess its practical impact. These concerns primarily affect the instance-level track; category-level queries are constructed directly from image–category annotations.
-
The currently released artifacts do not support exact reproduction of the reported results. The repository provides only a 10% query sample and describes the code as a reference implementation rather than a self-contained reproducible package. More importantly, the released grounding prompt explicitly differs from the frozen prompt used to obtain the paper’s metrics. The exact direct-baseline prompts and inference settings are also insufficiently documented. These gaps make it difficult to reproduce the comparisons or assess sensitivity to prompting, especially because Ref3DOL is a central contribution. The full evaluation manifests, evaluation code, and configurations corresponding to the reported tables would substantially strengthen the submission.
Questions for authors
-
How does the method compare with an adapted OVMono3D-GEO pipeline under matched grounding, segmentation, depth, and intrinsics? Could the authors isolate the contributions of gravity alignment and rectangle fitting, and clarify whether “without pose recovery” refits an axis-aligned box or merely resets the final rotation?
-
Could the authors independently audit a random sample of instance-level queries and report ambiguity and target-identification error rates? Could they also measure how often the camera-coordinate ordering used in query construction disagrees with the corresponding image-space ordering?
-
Could the authors provide the exact prompts and inference settings for both the proposed system and the direct baselines, together with evaluation code and complete query manifests or generation code? Which parts of the released reference implementation differ from those used to produce Tables 2–5?
-
Appendix B.5 attributes residual errors to instance masks, occlusion, gravity estimation, and box fitting. Could the authors further distinguish incomplete object extent from errors in these remaining components? In particular, what additional analysis would explain the 24.00% [email protected] achieved with both ground-truth 2D boxes and depth, given that the masks and gravity estimates remain predicted?