Towards Camera-Robust 3D Localization: Equation-Anchored Tool-Use for MLLMs
Abstract
3D localization in Multimodal Large Language Models (MLLMs), including 3D object detection and 3D visual grounding, is fundamentally limited by camera intrinsic ambiguity: the same image admits different 3D scenes under different cameras. Existing MLLMs either ignore camera parameters and overfit to a canonical training intrinsic, or retrieve depth and 3D cues from external tools but treat the returned values as reference cues (numerical hints that the model is free to interpret implicitly) rather than formula variables in an explicit geometric computation. We propose an equation-anchored tool-use framework that re-purposes spatial tools as formula variables. The proposed framework proactively retrieves camera intrinsics and samples multi-point metric depths, writes the pinhole back-projection equation explicitly in Chain-of-Thought (CoT), and substitutes the tool outputs into the formula to compute a geometry-derived 3D center estimate. The MLLM then combines this estimate with visual evidence to predict the final center, dimensions, and orientation of the 9-DoF bounding box. On both 3D object detection and 3D visual grounding tasks under rescaled camera intrinsics from to , our method outperforms RGB-only and tool-augmented baselines, with significant gains where the camera deviates most from the training scale. Code and data will be released.
est. 32% chance this paper gets accepted at ICLR 2027.
What do you think this paper will get?
All positions stay anonymous.