- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper studies camera-robust 3D object detection and visual grounding with multimodal large language models. The proposed framework first predicts 2D bounding boxes, retrieves camera intrinsics and object-level depth samples, and explicitly substitutes these quantities into the pinhole back-projection equations within a chain of thought. The resulting center estimate, together with visual information, conditions the final 9-DoF bounding-box prediction. The model is trained using structured tool-use and reasoning traces, and evaluated across camera intrinsic rescaling factors from 0.5× to 1.5×.
Critical strengths
The framework achieves improvements over the evaluated RGB-only baselines across most rescaling settings, including when using predicted rather than ground-truth depth. For example, at 1.0×, the 8B model improves detection performance from 38.20 to 44.13; at 1.5×, grounding accuracy improves from 20.69 to 28.12. The intermediate tool outputs and geometric estimates also make the pipeline easier to inspect.
Critical weaknesses
-
My main concern is that the paper has not established that its gains exceed those obtainable from adding geometric information and applying standard back-projection. Camera intrinsics and metric depth provide information directly relevant to absolute 3D localization, while pinhole back-projection is a standard geometric operation. Expressing this computation as a textual chain of thought does not, by itself, establish a substantial methodological contribution. The comparisons do not sufficiently isolate the value of equation-anchored reasoning. The main RGB-only baselines lack the geometric inputs available to the proposed method, and the complete training pipeline additionally introduces intermediate 2D localization and reasoning supervision. Consequently, the observed gains could arise from richer inputs and additional supervision rather than the proposed reasoning mechanism. Table 5 does not clearly establish whether the no-CoT baseline retains identical geometric inputs and intermediate supervision, while the prompting experiments without task-specific fine-tuning do not resolve this issue. A crucial missing baseline is a pipeline that uses the same 2D predictions, intrinsics, and depth samples, computes the center estimate deterministically, and supplies it to an equally trained downstream model. Without this comparison, it remains unclear why the standard calculation should be generated within an MLLM’s chain of thought, or what benefit this provides over a simpler implementation. The reported results support the usefulness of the overall geometry-assisted pipeline, but do not yet substantiate the claimed methodological advance.
-
Insufficiently specified camera rescaling protocol. The paper does not clearly explain how RGB images, depth maps, intrinsics, image dimensions, and evaluated objects are transformed together. The padding and cropping visible in Figures 7–8 may confound camera-induced geometric errors with changes in object resolution, context, and visibility, requiring clearer specifications and controlled experiments.
-
Limited evidence for camera robustness and cross-dataset generalization. At 0.5×, grounding accuracy falls from 55.17 to 13.26, below VG LLM’s 17.51; using ground-truth depth yields only 11.41, so depth-estimation errors alone cannot explain the degradation. The SUN RGB-D experiment uses 5,000 additional intrinsic-adjusted ScanNet examples for LoRA adaptation without equally adapted baselines, leaving the framework’s independent transfer advantage unclear. Comparing pixel focal lengths alone also inadequately characterizes camera differences because these values depend on image resolution.
-
Overstated preservation of general capabilities. The appendix claims comparable general performance, but Table 11 shows MMLU-Pro dropping from 66.77 to 44.63 and the six-benchmark average declining from 61.26 to 57.03. The authors should acknowledge this degradation and discuss the trade-off between specialized improvements and general capabilities.
Questions for authors
-
Can you compare against a baseline that computes back-projection deterministically and supplies the resulting center estimate to the same downstream model, using matched inputs, training data, and training budgets?
-
In Table 5, does the no-CoT baseline receive the same camera intrinsics, depth samples, and intermediate 2D supervision? If not, can you provide a controlled ablation that isolates explicit equation-based reasoning?
-
Can you compare equally trained models that receive geometric quantities as contextual inputs, receive a programmatically computed center, or explicitly generate the calculation in CoT, across the full rescaling range?
-
What evidence shows that the generated calculations influence the final predictions? Please report calculation accuracy and controlled interventions on the geometric inputs or intermediate center estimates.