acceptodds
Withdrawn from ICLR 2027

Training-Free Referential Monocular 3D Object Localization

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

We study referential monocular 3D object localization (RM3DOL): localizing a language-referred object as a metric camera-frame 9-DoF box from a single RGB image. The task couples fine-grained grounding with metric recovery under monocular depth and scale ambiguity. We propose a training-free framework that grounds the target semantically, independently reconstructs a gravity-aligned metric representation, and combines both outputs to recover position, dimensions, and orientation. We also introduce Ref3DOL, an indoor benchmark built from Omni3D-unified SUN RGB-D and Hypersim annotations. Its automated progressive-disambiguation pipeline uses category, appearance, and spatial cues to generate instance-specific queries without manual expression annotation. The category- and instance-level tracks contain 33,915 and 41,875 queries, respectively. Across multiple VLM grounding backbones, our framework achieves the best overall performance among the evaluated direct 3D-box prediction baselines, including Qwen3-family models and GPT-5.6 Sol. Code and data are available at https://anonymous.4open.science/r/TrainingFreeRM3DOL-9744/README.md.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Referential 3D Object Localization, Training-Free Monocular 3D Perception, Vision-Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_YjuuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies referential monocular 3D object localization: predicting a metric camera-frame 3D bounding box for an object specified by a natural-language query in a single RGB image. The proposed framework combines frozen VLM-based 2D grounding, instance segmentation, monocular metric geometry estimation, gravity estimation, point-cloud filtering, and gravity-constrained box fitting. It also introduces Ref3DOL, constructed from Omni3D-unified SUN RGB-D and Hypersim annotations, with 33,915 category-level and 41,875 instance-level queries. The best configuration outperforms the evaluated direct 3D-box prediction baselines, while ablations and oracle experiments examine several sources of error.

Critical strengths
  1. A practical framework with measurable gains over the evaluated direct predictors. The method requires no task-specific parameter updates and supports several grounding backbones. On the instance-level track, its best configuration improves [email protected] over the strongest evaluated direct baseline by 5.15 points on SUN RGB-D and 9.33 points on Hypersim. These results support the usefulness of composing pretrained semantic and geometric capabilities as an alternative inference pipeline.

  2. Useful component and oracle analyses. The grounding and segmentation ablations show the importance of resolving the referred instance and isolating its pixels. On SUN RGB-D with ground-truth intrinsics, replacing predicted depth with ground-truth depth raises [email protected] from 35.25 to 60.67, compared with 43.01 when replacing only the 2D box. These experiments provide actionable evidence that metric depth is a major bottleneck in the evaluated system.

Critical weaknesses
  1. The methodological contribution relative to closely related geometric pipelines is insufficiently established. OVMono3D-GEO already combines pretrained 2D detection, segmentation, metric depth, geometric unprojection, and point-cloud processing without task-specific training. The pseudo-box generation stage of OVM3D-Det also includes geometry-based box construction and ground-aligned orientation estimation, although its complete framework subsequently trains a detector. These related pipelines are not discussed. Their open-vocabulary detection setting differs from referential localization, but their geometric components provide relevant points of comparison. The current comparisons against direct VLM predictions do not isolate the contribution of the proposed reconstruction and fitting design from the benefit of using dedicated pretrained geometry models. An adapted geometric baseline with matched grounding, masks, depth, and intrinsics would help establish the incremental contribution.

  2. The generated instance-level queries need stronger independent validation. Attribute extraction and the principal automatic verification both use Qwen3-VL-32B. The reported manual inspection of incorrect selections is useful, but it does not independently audit a random sample of all queries or exclude errors shared by the generation and verification stages. Appendix A also maps camera-frame X/Y coordinate order to image-relative ordering expressions. Under perspective projection, image position depends on X/Z and Y/Z, so the stated ordering rules do not guarantee consistency with image-space order when depths differ. This identifies a potential failure mode rather than demonstrating its prevalence in Ref3DOL. An independent query audit and a projection-consistency analysis are needed to assess its practical impact. These concerns primarily affect the instance-level track; category-level queries are constructed directly from image–category annotations.

  3. The currently released artifacts do not support exact reproduction of the reported results. The repository provides only a 10% query sample and describes the code as a reference implementation rather than a self-contained reproducible package. More importantly, the released grounding prompt explicitly differs from the frozen prompt used to obtain the paper’s metrics. The exact direct-baseline prompts and inference settings are also insufficiently documented. These gaps make it difficult to reproduce the comparisons or assess sensitivity to prompting, especially because Ref3DOL is a central contribution. The full evaluation manifests, evaluation code, and configurations corresponding to the reported tables would substantially strengthen the submission.

Questions for authors
  1. How does the method compare with an adapted OVMono3D-GEO pipeline under matched grounding, segmentation, depth, and intrinsics? Could the authors isolate the contributions of gravity alignment and rectangle fitting, and clarify whether “without pose recovery” refits an axis-aligned box or merely resets the final rotation?

  2. Could the authors independently audit a random sample of instance-level queries and report ambiguity and target-identification error rates? Could they also measure how often the camera-coordinate ordering used in query construction disagrees with the corresponding image-space ordering?

  3. Could the authors provide the exact prompts and inference settings for both the proposed system and the direct baselines, together with evaluation code and complete query manifests or generation code? Which parts of the released reference implementation differ from those used to produce Tables 2–5?

  4. Appendix B.5 attributes residual errors to instance masks, occlusion, gravity estimation, and box fitting. Could the authors further distinguish incomplete object extent from errors in these remaining components? In particular, what additional analysis would explain the 24.00% [email protected] achieved with both ground-truth 2D boxes and depth, given that the masks and gravity estimates remain predicted?

Reviewer_dAADOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The authors propose a training-free framework to handle RM3DOL task. The authors also introduce a benchmark Ref3DOL from SUN RGB-D and Hypersim datasets to support the evaluation. The proposed framework is validated across several VLM grounding backbones and outperform existing VLM baselines.

Critical strengths
  1. The paper introduce a useful indoor benchmark for the RM3DOL task by constructing the query via progressive disambiguation. The proposed benchmark could be potentially used by later research.
Critical weaknesses
  1. The novelty of the proposed method is limited. The proposed training-free semantic-geometric duo branch is very similar to OVMono3D-GEO proposed in [A]. Most components are existing pretrained models or standard geometric operations. Therefore the claimed novelty is limited. The authors need to adapt and compare against such training-free methods on 3D grounding. Currently it is unclear what is the core new contribution of the method, and how it is different from prior work.
  2. The experiments on the benchmark validation and model performance are insufficient. To the reviewer's understanding, the benchmark primarily contributes the generated queries over existing annotations. The quality of these queries are not independently established through human evaluation. The VLM automated check does not verify the quality of the queries, such as whether they are natural and semantically correct. For the main experiments in Tab. 2, the comparison is only against general-purpose VLMs performing direct 3D prediction. The authors should compare stronger 3D grounding specialists and modular 2D-to-3D architectures (such as OVMono3D-GEO).

[A] "Open Vocabulary Monocular 3D Object Detection", 3DV 2026.

Questions for authors
  1. Based on weakness 1, could the authors clarify how the proposed framework differ from OVMono3D-GEO, which also uses 2D grounding, segmentation, metric depth, DBSCAN and 3D box lifting?
  2. Based on weakness 2, could the authors compare with more existing 3D grounding specialists and modular 2D-to-3D architectures?
  3. The proposed RM3DOL task seems similar to the existing monocular 3D visual grounding task introduced by [B]. In the introduction it is not very clear whether the authors are claiming a new task (with 9-DoF annotations and referring expressions) or not. Please clarify and distinguish with the prior work.
  4. From Tab.2, the performance gain on SUN RGB-D @0.5 is quite small for both levels (e.g. 8.38 over 8.17 for category-level). Could the authors discuss some insights on why this is the case?

[B] "Mono3DVG: 3D Visual Grounding in Monocular Images", AAAI 2024.

Reviewer_s45rOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies referential monocular 3D object localization where the goal is to predict a metric 9 DoF bounding box for a language referred object from a single RGB image. The authors propose a training free pipeline that separates semantic grounding from geometric reconstruction. A VLM first identifies the target in 2D followed by SAM 3 segmentation. Separately the method uses pretrained metric depth estimation and camera calibration models to reconstruct 3D points. The target points are filtered using DBSCAN and fitted with a gravity constrained bounding box.

The paper also introduces Ref3DOL, a benchmark constructed from SUN RGB-D and Hypersim using Omni3D annotations. Experiments show improvements over several direct VLM based 3D box prediction baselines. The best method achieves mAcc @0.25 of 34.16% on SUN RGB-D and 26.74% on Hypersim for instance grounding. Additional ablations and oracle experiments examine the contribution of different components and identify metric depth estimation as a major bottleneck.

Critical strengths

This works looks like a potentially useful benchmark for indoor referential 3D localization: Ref3DOL provides a reasonably large evaluation set with both category and instance level queries across real and synthetic indoor scenes. The automatic query construction pipeline could also make it easier to extend the benchmark to other datasets. This is probably the strongest contribution of the paper assuming the generated queries are sufficiently accurate and challenging.

Also they present a consistent improvements over the evaluated direct prediction baselines.The proposed pipeline performs better than the strongest tested direct VLM baselines at IoU 0.25 on both datasets. The authors also evaluate multiple grounding backbones and provide useful ablations and oracle experiments. I especially appreciate the analysis showing that depth estimation remains a large bottleneck. These experiments make the work more useful than simply proposing a combination of existing models.

Critical weaknesses

First concern is on the novelty. The proposed architecture is very close to existing training free monocular 3D detection pipelines. In particular OVMono3D GEO already combines 2D localization, segmentation, metric depth, point cloud reconstruction, DBSCAN and geometric box fitting. The paper by Kamali follows a similar training free approach. The main differences here appear to be the use of a VLM for referring expressions, newer pretrained components and gravity constrained rectangle fitting. These are reasonable design choices but I am not convinced they establish enough methodological novelty for ICLR. The paper does not compare against OVMono3D-GEO or clearly explain what is technically new relative to it.

The absolute performance also remains quite low at higher IoU thresholds. For instance the best method achieves only 8.16% mAcc @0.50 on SUN RGB-D instance localization compared with 7.54% for the strongest direct baseline. This is a fairly small gain and makes it difficult to conclude that the method recovers accurate 3D geometry. The Qwen2.5-VL 3B variant also performs worse than the strongest direct baseline on SUN RGB-D instance localization at IoU 0.25.

I also have concerns about the benchmark generation. The authors use Qwen3-VL-32B to extract attributes and then use the same model to validate the generated referring expressions. This is not an independent validation and could introduce some bias. The paper does not report enough details about the percentage of genuinely ambiguous examples or how often the different disambiguation stages are needed. Queries that use annotated 3D ordering may also contain relationships that are difficult to infer reliably from a single RGB image.

Questions for authors
  1. How does the proposed approach compare against OVMono3D-GEO under the same Ref3DOL evaluation? It would be useful to see a comparison between PCA based geometric fitting and the proposed gravity constrained box fitting while keeping the grounding, segmentation and depth components fixed. Without this comparison it's hard to tell how much of the improvement comes from the actual method versus simply using stronger pretrained models.

2.It's not clear how many of the instance level queries contain multiple objects of the same category. Can the authors report accuracy separately for unique and multiple instance cases? It would also be useful to break down the results based on color, absolute position and relative ordering to better understand where the benchmark is challenging. I'm also a bit concerned about the query validation process since Qwen3-VL-32B is involved in both generating and validating the queries. The reported validation accuracy alone doesnt seem sufficient to rule out ambiguity or bias. Some independent human evaluation of query correctness would help here.

3)The geometric fitting relies on visible surface points but it's not clear how the method recovers the full physical dimensions of objects that are partially occluded or have large unseen surfaces. The oracle analysis is interesting here since even with ground truth 2D boxes and depth the performance still seems quite limited on SUN RGB-D at stricter IoU thresholds. Can the authors provide a better breakdown of the remaining errors from segmentation, occlusion and geometric fitting? Also while the output has nine parameters the orientation is constrained by the estimated gravity direction and a single heading angle. I'm curious how well this assumption holds for objects that are tilted or dont follow the assumed upright geometry.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.