acceptodds
Withdrawn from ICLR 2027

Towards Camera-Robust 3D Localization: Equation-Anchored Tool-Use for MLLMs

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

3D localization in Multimodal Large Language Models (MLLMs), including 3D object detection and 3D visual grounding, is fundamentally limited by camera intrinsic ambiguity: the same image admits different 3D scenes under different cameras. Existing MLLMs either ignore camera parameters and overfit to a canonical training intrinsic, or retrieve depth and 3D cues from external tools but treat the returned values as reference cues (numerical hints that the model is free to interpret implicitly) rather than formula variables in an explicit geometric computation. We propose an equation-anchored tool-use framework that re-purposes spatial tools as formula variables. The proposed framework proactively retrieves camera intrinsics and samples multi-point metric depths, writes the pinhole back-projection equation explicitly in Chain-of-Thought (CoT), and substitutes the tool outputs into the formula to compute a geometry-derived 3D center estimate. The MLLM then combines this estimate with visual evidence to predict the final center, dimensions, and orientation of the 9-DoF bounding box. On both 3D object detection and 3D visual grounding tasks under rescaled camera intrinsics from to , our method outperforms RGB-only and tool-augmented baselines, with significant gains where the camera deviates most from the training scale. Code and data will be released.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D Localization, Multimodal Large Language Models, Tool-Augmented Reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_sKt1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies camera-robust 3D object detection and visual grounding with multimodal large language models. The proposed framework first predicts 2D bounding boxes, retrieves camera intrinsics and object-level depth samples, and explicitly substitutes these quantities into the pinhole back-projection equations within a chain of thought. The resulting center estimate, together with visual information, conditions the final 9-DoF bounding-box prediction. The model is trained using structured tool-use and reasoning traces, and evaluated across camera intrinsic rescaling factors from 0.5× to 1.5×.

Critical strengths

The framework achieves improvements over the evaluated RGB-only baselines across most rescaling settings, including when using predicted rather than ground-truth depth. For example, at 1.0×, the 8B model improves detection performance from 38.20 to 44.13; at 1.5×, grounding accuracy improves from 20.69 to 28.12. The intermediate tool outputs and geometric estimates also make the pipeline easier to inspect.

Critical weaknesses
  1. My main concern is that the paper has not established that its gains exceed those obtainable from adding geometric information and applying standard back-projection. Camera intrinsics and metric depth provide information directly relevant to absolute 3D localization, while pinhole back-projection is a standard geometric operation. Expressing this computation as a textual chain of thought does not, by itself, establish a substantial methodological contribution. The comparisons do not sufficiently isolate the value of equation-anchored reasoning. The main RGB-only baselines lack the geometric inputs available to the proposed method, and the complete training pipeline additionally introduces intermediate 2D localization and reasoning supervision. Consequently, the observed gains could arise from richer inputs and additional supervision rather than the proposed reasoning mechanism. Table 5 does not clearly establish whether the no-CoT baseline retains identical geometric inputs and intermediate supervision, while the prompting experiments without task-specific fine-tuning do not resolve this issue. A crucial missing baseline is a pipeline that uses the same 2D predictions, intrinsics, and depth samples, computes the center estimate deterministically, and supplies it to an equally trained downstream model. Without this comparison, it remains unclear why the standard calculation should be generated within an MLLM’s chain of thought, or what benefit this provides over a simpler implementation. The reported results support the usefulness of the overall geometry-assisted pipeline, but do not yet substantiate the claimed methodological advance.

  2. Insufficiently specified camera rescaling protocol. The paper does not clearly explain how RGB images, depth maps, intrinsics, image dimensions, and evaluated objects are transformed together. The padding and cropping visible in Figures 7–8 may confound camera-induced geometric errors with changes in object resolution, context, and visibility, requiring clearer specifications and controlled experiments.

  3. Limited evidence for camera robustness and cross-dataset generalization. At 0.5×, grounding accuracy falls from 55.17 to 13.26, below VG LLM’s 17.51; using ground-truth depth yields only 11.41, so depth-estimation errors alone cannot explain the degradation. The SUN RGB-D experiment uses 5,000 additional intrinsic-adjusted ScanNet examples for LoRA adaptation without equally adapted baselines, leaving the framework’s independent transfer advantage unclear. Comparing pixel focal lengths alone also inadequately characterizes camera differences because these values depend on image resolution.

  4. Overstated preservation of general capabilities. The appendix claims comparable general performance, but Table 11 shows MMLU-Pro dropping from 66.77 to 44.63 and the six-benchmark average declining from 61.26 to 57.03. The authors should acknowledge this degradation and discuss the trade-off between specialized improvements and general capabilities.

Questions for authors
  1. Can you compare against a baseline that computes back-projection deterministically and supplies the resulting center estimate to the same downstream model, using matched inputs, training data, and training budgets?

  2. In Table 5, does the no-CoT baseline receive the same camera intrinsics, depth samples, and intermediate 2D supervision? If not, can you provide a controlled ablation that isolates explicit equation-based reasoning?

  3. Can you compare equally trained models that receive geometric quantities as contextual inputs, receive a programmatically computed center, or explicitly generate the calculation in CoT, across the full rescaling range?

  4. What evidence shows that the generated calculations influence the final predictions? Please report calculation accuracy and controlled interventions on the geometric inputs or intermediate center estimates.

Reviewer_PgvROpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper fine-tunes Qwen3-VL-8B to do single-image 3D detection and 3D visual grounding through a fixed multi-turn trace: predict 2D boxes, call a tool that returns the dataset intrinsics, call a tool that returns five depth samples inside a SAM3 mask (ground-truth depth or UniDepthV2), write the pinhole back-projection of the 2D box center at the mean sampled depth in the chain of thought, then output the 9-DoF box. Training traces come from ScanNet frames for detection and from ScanRefer frames whose expressions were regenerated and filtered by Qwen3.5-Plus for grounding. With intrinsics rescaled from 0.5x to 1.5x, the model beats an RGB-only Qwen3-VL SFT baseline and VG LLM at most factors (44.13 vs 38.20 [email protected] over 31 classes at 1.0x). The appendix adds ablations at 1.0x, a 4B comparison with an adapted Think3D, and a LoRA transfer to SUN RGB-D.

Critical strengths

The starting point is correct and clearly stated: a single image does not fix metric 3D without the camera, so the intrinsics have to enter the computation somewhere. Tables 6 and 7 are a useful negative result. Putting the same intrinsics, depth samples, and even the equation into the prompt of an untuned model (Qwen3-VL-8B, GPT-6 Astra) does not help consistently and often hurts (for Qwen3-VL-8B, detection falls from 16.02 to 9.73 once the tool outputs are added), so supervision on how to use the numbers matters.

Critical weaknesses

The central claim is never tested. The paper argues that tool outputs work better as formula variables than as reference cues, so the decisive control is the same SFT model given the same intrinsics and depth samples but trained to output the box without the equation step. That run is missing. The "w/o CoT" row of Table 5 reproduces the RGB-only Qwen3-VL SFT baseline exactly (38.20 on 31-class detection and 53.85 on grounding, cf. Tables 1-2), so it removes the tools together with the equation. Table 3 varies which tools are available, but no row gives the model both tools without the equation step, and the only reference-cue comparisons are untuned prompting (Tables 6-7) and a Think3D adaptation that receives rendered views rather than K and depth. Every ablation is at 1.0x, so none of them says anything about camera robustness. As presented, all of the gains can be explained by the method receiving ground-truth intrinsics and metric depth that no baseline receives, which A.2 concedes for the generalist models. A deterministic pipeline (predicted 2D box, SAM3 mask, median depth, back-projection with K) and a calculator tool in place of LLM arithmetic would also show what the language model itself contributes.

The evaluation does not support "camera-robust". The rescaling protocol is not defined in the text. Figs. 7 and 9 show that factors below 1 shrink the image and pad it with black, and factors above 1 crop and upsample, so half of the range is low resolution plus a black border that no real camera produces; both failure cases in Fig. 9 are SAM3 masks spilling into that border. Even on this protocol the method degrades almost as fast as the baseline: detection falls from 44.13 to 16.97 at 0.5x (-62%, against -72% for Qwen3-VL), VG LLM beats both of your variants in grounding at 0.5x (17.51 vs 13.26 and 11.41) and the UniDepth variant in detection at 1.4x (24.07 vs 21.65), which contradicts "significant gains where the camera deviates most". The 4B experiments are weaker still: with predicted depth, the method is below the RGB-only Qwen3-VL-4B baseline at 1.0x on both tasks (40.18 vs 40.70 detection, 54.64 vs 57.03 grounding, Tables 8-9), which the text does not mention. The only real cross-camera test, SUN RGB-D, needs LoRA on ScanNet images resized to SUN RGB-D intrinsics, reports no baseline, and costs 4.7 points on ScanNet at 1.0x. Camera-aware monocular 3D detection is an established line that is neither cited nor compared, e.g., virtual depth in Cube R-CNN (Brazil et al., CVPR 2023) and the canonical camera transform of Metric3D (Yin et al., ICCV 2023). And once K and metric depth are given, as in the GT-depth variant, the task is single-view RGB-D detection, where geometric detectors are invariant to intrinsic scaling by construction.

Questions for authors
  1. (Requested experiment; this would change my assessment) The SFT model with identical tool outputs in its trace but no equation step, evaluated across the rescale factors and not only at 1.0x.
  2. How is a rescale factor applied to the image, the intrinsics, the depth map, and the ground truth? Are objects that leave the cropped field of view at factors above 1 kept in the ground truth?
  3. How often does the center written in the CoT match the exact back-projection, and how far is the final predicted center from it? If the final center often departs from the computed one, "equation-anchored" overstates what the model does.
  4. (Requested analysis) An error breakdown across scales: 2D box accuracy, error of the geometry-derived center, final center error, size and orientation error. With GT depth, the drop from 44.90 to 18.93 at 0.5x is not explained by depth quality.
Reviewer_F9ZfOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

In this paper, the author propose a tool-call based 3D grounding with multimodal LLMs. The approach formulates 3D grounding as a tool-assisted reasoning problem, where the model is trained with explicit CoT to acquire camera intrinsics and metric depth with tool calls, and treat them as context for final prediction. Experiments were conducted on ScanNet, ScanRefer and SUN RGB-D with varying rescale factors.

Critical strengths
  • The paper is generally easy to understand, even though some details and terms were not disclosed properly.

  • The proposed method is generally reasonable and could be used as an engineering approach for VLM-based 3D grounding/detection systems in practice.

Critical weaknesses

I consider that the paper have significant issues to be addressed, which makes it below the acceptance bar for ICLR:

  • First, the positioning of the paper is problematic. The proposed method describes itself as a tool-assisted CoT one, but the actual pipeline is rather straightforward. The intrinsics tool is basically just reading the parameters provided by the dataset, while the acquire of 2D object information, metric depth and segmentation are also standardized VLM prompting pipelines. The method adopts a fixed multi-turn format rather than learning from actual multi-turn agentic training. This makes it feels like that the novelty and story of this work are created ones. The direct reading of given intrinsics instead of tool-based estimation significantly limits the practical value of the work.

  • Underneath the surface of explicit 3D geometry reasoning, Equation 4 is actually calculated using CoT instead of external calculator. No verification mechanism is imposed, so writing the equation explicitly does not guarantee correct computation. In addition, Equation 4 is used as a context for the VLM to finally predict the final 3D grounding output. This is nothing different from an implicit way to use quantified 3D cues, which contradicts with the claims that distinguish the paper from other related works.

  • A question regarding Table 5 - It is unclear what the baseline setting is and whether the baseline receives the same tool outputs (or rather, directly predicts 3D boxes from scratch). Thus, it's unclear whether the gains come from explicitly evaluating the equation in CoT or from the additional geometric inputs and intermediate supervision. A useful comparison would provide the VLM with the same information and a reference XYZ computed in code, to isolate the benefit of generating the calculation in CoT.

  • The paper misses relevant work such as LocateAnything3D and WildDet3D. LocateAnything3D’s Chain-of-Sight is seems related to its use of intermediate predictions (especially starting with 2D object), while WildDet3D offers broader evaluation across datasets and camera setups. The rescaling experiments are useful and makes sense as ablation studies, but stronger comparisons on established benchmarks and protocols with recent works are needed to put the gains in context.

  • I also feel that the rescale factor setup in the paper is an over-simplification of the robustness evaluation. In real world application, changes in camera intrinsics are often not simple rescaling. This is reflected on the results of the experiment, where the proposed approach does not show decisive gains over baselines on unseen scales. Unlike many 3D approaches that often fail catastrophically on unseen camera intrinsics, the improvements in the table are pretty stable over different scales. This shows that the setting fails to really challenge the true camera robustness and that it is not clear whether the improvement comes from training with in-domain data.

  • The training/evaluation split is unclear. The paper identifies ScanNet and ScanRefer as data sources, but does not clearly specify the partitions, sample counts, or whether scenes are disjoint between training and evaluation. This limits reproducibility of the work.

Questions for authors

Please refer to the weakness section.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.