acceptodds
Withdrawn from ICLR 2027

Where Should I Stand? Predicting Camera Pose from Perspective Descriptions for Spatial Reasoning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multi-image spatial reasoning remains challenging for current multimodal large language models (MLLMs). While single-view perception is inherently 2D, reasoning over multiple views requires building a coherent scene understanding across viewpoints. In particular, we study perspective taking, where a model must build a coherent 3D understanding from multi-view observations and use it to reason from a new, language-specified viewpoint. We introduce CAMCUE, a pose-aware multi-image framework that uses camera pose as an explicit geometric anchor for cross-view fusion and novel-view reasoning. CAMCUE injects per-view pose into visual tokens, grounds natural-language viewpoint descriptions to a target camera pose, and synthesizes a pose-conditioned imagined target view to support answering. To support this setting, we curate CAMCUE-DATA with 27,668 training and 508 test instances pairing multi-view images and poses with diverse target-viewpoint descriptions and perspective-shift questions. We also include human-annotated viewpoint descriptions in the test split to evaluate generalization to human language. CAMCUE improves overall accuracy by 9.06% and predicts target poses from natural-language viewpoint descriptions with over 90% rotation accuracy within 20° and translation accuracy within a 0.5 error threshold. This direct grounding avoids expensive test-time search-and-match, reducing inference time from 256.6s to 1.45s per example and enabling fast, interactive use in real-world scenarios.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
large language model, multimodal large language models, spatial reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_REFDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CAMCUE, a framework for perspective-taking spatial reasoning with MLLMs. CAMCUE injects camera-pose information from the contextual views into visual representations and trains the MLLM to jointly answer a given question and predict the camera pose corresponding to the language-specified target viewpoint. At inference time, the predicted target pose is used to synthesize an imagined target-view image, which is then provided to the MLLM as additional visual evidence for final answer prediction. Experimental results show that CAMCUE outperforms the corresponding MLLM backbones, a world-model-based test-time scaling method (MindJourney), and pose-free target-view generation baselines.

Critical strengths

The idea of explicitly grounding a language-specified target viewpoint into a camera pose and conditioning novel-view synthesis on that pose is intuitively well motivated. In Tables 2 and 4, CAMCUE outperforms direct MLLM inference, a search-based world-model approach, and methods that directly synthesize the target-view image from language and reference images without explicit target-pose grounding. These results support the effectiveness of explicit viewpoint grounding and pose-conditioned image synthesis for multi-image perspective-taking QA, and I consider this a meaningful contribution to this area.

Critical weaknesses
  1. Limited validation under unavailable or noisy camera parameters: The main CAMCUE inference pipeline assumes access to the camera intrinsics and extrinsics of the contextual images. This assumption is reasonable in robotic or scanning environments where camera calibration and localization are available, but may not hold for generic collections of images. Moreover, in practical settings, these camera parameters may need to be estimated and can therefore contain non-negligible errors. The paper neither evaluates CAMCUE using estimated camera parameters nor analyzes its robustness to perturbations in these parameters. As a result, the range of real-world settings in which the method can be reliably applied remains somewhat unclear.
  2. The necessity of joint multi-task training and the source of the performance gain are not fully disentangled: The central question suggested by the Introduction is whether multi-image perspective-taking QA can be improved by explicitly mapping a language-specified viewpoint to a camera pose and using a pose-conditioned novel view for reasoning. However, it is less clear why answer generation and target-pose prediction need to be jointly trained within the same MLLM. Figure 4 shows that the improvement without the imagined view is relatively modest, whereas a much larger gain appears once the synthesized target view is provided. This leaves open the possibility that a simpler pipeline, where training a target-pose predictor, synthesizing an image from the predicted pose, and providing that image to an otherwise unchanged MLLM, could achieve a similar benefit. At the same time, the current experiments are insufficient to conclude that such a simpler pipeline would be equally effective. I would therefore consider that the experimental evidence in the Experimental section or the strong motivation in the Introduction section, that support the necessity of multi-task training.
Questions for authors

Note that the following points are not primary factors in my rating, but I believe addressing them would improve the clarity and positioning of the paper.

  1. [Clarification] Consistency between the Pose Fusion diagram in Fig. 2 and Eq. 3: Eq. 3 indicates that the image and camera features are concatenated, passed through an MLP/projection, and then added to the original image features through a residual connection. However, in the enlarged Pose Fusion diagram in Fig. 2, the output of the Fusion MLP appears to be routed back toward the input side, and the distinction between concatenation and addition is visually unclear. Is the intended computation exactly the one described in Eq. 3? I suggest clarifying the connections and operators in the figure.

  2. [Suggestion] Positioning with respect to recent related work: I encourage the authors to expand the discussion in the Introduction and Related Work regarding recent (i.e., 2026) work on cross-view/multi-view spatial reasoning and spatial reasoning using intermediate generated images or test-time visual imagination. For example, relevant recent work includes, but is not limited to: [1,2,3]

[1] Yang et al., How and What to Imagine? Visual Thinking in Unified Multimodal Models for Cross-View Spatial Reasoning, EMNLP2026.

[2] Oi et al., From Correspondence to Actions: Human-Like Multi-Image Spatial Reasoning in Multi-modal Large Language Models, ICML2026.

[3] Yu et al., When and How Much to Imagine: Adaptive Test-Time Scaling with World Models for Visual Spatial Reasoning, arXiv2026.

Reviewer_dSzJOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces the CamCue framework and the CamCue-Data dataset. Given multiple contextual images and their camera parameters, CamCue fuses Plücker ray codes into visual features, predicts the target camera pose based on a textual description, and then uses LVSM to synthesize images from the target viewpoint to assist the model in answering questions. The paper also introduces CamCue-Data, which contains 27,668 training question-answer pairs and 508 test question-answer pairs. Experiments show that this framework improves the question-answering accuracy on our self-developed benchmark across three MLLM.

Critical strengths
  1. The research question is of significant value. Understanding scenes from a new perspective specified by language is a key challenge in multi-image spatial reasoning. This paper provides a clear and interpretable solution by explicitly linking language descriptions, cross-view geometry, and target view synthesis through camera pose.

  2. The overall framework achieves consistent improvements across multiple multimodal LLM. On CamCue-Data, CamCue increases the overall accuracy of Qwen2.5-VL-7B, Qwen2.5-VL-3B, and InternVL2.5-8B by 9.06, 8.40, and 9.25 percentage points, respectively.

Critical weaknesses
  1. There is a lack of sufficient comparative experiments. The paper combines spatial question-answering, object pose prediction, and image generation, but I am very unsure which of these constitutes the paper’s core contribution. The complete workflow was primarily evaluated on the authors’ self-built CamCue-Data dataset, and comparisons with representative existing spatial reasoning methods were also rather limited.

  2. The novelty of the method requires more thorough justification. The use of generated views as auxiliary evidence to improve spatial reasoning has already been explored in works such as MindJourney. The potential contribution of this paper lies in mapping linguistic descriptions to explicit object pose and combining them with multi-view representations. However, the existing comparative and ablation experiments have not yet sufficiently demonstrated the independent value of these specific design choices.

  3. The effectiveness of the generated images has not been fully validated. The paper provides qualitative image comparisons and downstream question-answering results, but this evidence is insufficient to demonstrate whether predicting pose can yield images that are more accurate and better aligned with the specified viewpoint, or how the images feed back into the question-answering task.

Questions for authors

See Weaknesses.

My main concern stems from issues with the comparative experiments; the current lack of comparisons with mainstream spatial intelligence datasets prevents me from clearly assessing the specific capabilities of this method.

Reviewer_7jHWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CamCue predicts a target camera pose from multiple scene images and a natural-language viewpoint description, then uses LVSM to synthesize a target view for spatial question answering. The authors introduce a corresponding dataset and evaluate the method with three vision-language backbones.

Critical strengths

I find the explicit target pose a sensible design choice: it gives viewpoint reasoning an intermediate prediction that can be inspected. The results also show practical gains. All three backbones improve in Table 2, and adding the synthesized view increases accuracy from 72.4% to 80.1% in Figure 4(a). This supports the usefulness of target-view feedback in the evaluated setting.

Critical weaknesses

My main question is whether the predicted poses are sufficiently accurate, and how that accuracy affects answering. For synthetic descriptions, Table 3 reports 91.5% at R@20°, but only 19.3% at R@5°. Comparing predicted and ground-truth poses with the same LVSM and QA model would help separate errors from pose prediction and image synthesis.

The camera-feature fusion design appears close to existing work. For example, RayZer also uses Plücker encoding, patch alignment, and concatenation followed by projection. CamCue differs in its residual formulation and application, but the additional technical contribution of this component remains unclear. A simple global camera embedding baseline within the complete pipeline would help establish what the patch-level design adds.

Some details of the renderer and baseline setup need clarification. The paper does not clearly specify the LVSM checkpoint or whether it was fine-tuned. Also, the Nano Banana experiments evaluate image-assisted QA, rather than GPT or Gemini answering the questions directly. Including one representative strong multimodal model as a direct QA baseline would better contextualize the benefit of the proposed pipeline.

Performance under less favorable inputs remains unclear. Appendix A uses four context images and requires them to collectively cover at least 80% of the target’s visible region. Results with noisy input poses and fewer context images would be useful, as would a breakdown by the actual magnitude of viewpoint change. These tests would help establish the conditions under which the method remains reliable.

One implementation detail also needs clarification: are the predicted rotation matrices made valid through parameterization or orthogonalization before error evaluation and view synthesis? The manuscript does not currently explain this, although that does not necessarily indicate an implementation error.

Questions for authors

See the weakness.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.