acceptodds
Withdrawn from ICLR 2027

OrientSAM: Mitigating Camera-Centric Shortcut in Multimodal Spatial Reasoning via Orientation-Aware Spatial Alignment

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multimodal large language models (MLLMs) still struggle with spatial reasoning that requires perspective transformation. Their predictions often favor camera-centric cues over the reference object's viewpoint, leading to systematic errors in non-camera reference settings. We analyze this behavior on binary left/right conflict samples and find that camera-centric answer preference varies with reference-object orientation. To address this issue, we propose OrientSAM, an orientation-aware spatial alignment framework for multimodal models. OrientSAM injects explicit orientation information into multimodal representations through orientation-aware tokens and Fourier-based angle encoding, and further adopts a curriculum learning strategy to progressively improve perspective-aware reasoning. In addition, we build a spatial data construction pipeline to generate orientation-aware spatial supervision from large-scale images. Experiments on Spatial-MM, ViewSpatial, and 3DSRBench show that OrientSAM delivers its strongest gains on non-camera-view, person-centric, and orientation-sensitive tasks. Paired conflict-subset comparisons show reduced camera-centric answer preference, supporting explicit orientation-aware spatial inputs for allocentric reasoning.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Orientation-Aware Spatial Alignment, Multimodal Spatial Reasoning, Camera-Centric Shortcut

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_vecDOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that MLLMs’ limited ability in reference-centric spatial reasoning is due to a camera-centric shortcut and proposes OrientSAM to mitigate it. The authors categorize samples as consistent or conflicting based on whether the camera-centric and reference-centric answers are the same, and show that accuracy is higher for consistent samples than for conflicting samples. OrientSAM uses external expert models to construct offline training supervision and to supply depth and orientation cues at inference, and integrates RGB, depth, and Fourier-encoded orientation features through structured multimodal tokens. The paper shows that OrientSAM achieves the highest accuracy on Spatial-MM non-camera-view samples among tested models, surpassing the existing APC-vis method by 0.74 percentage points while maintaining competitive camera-view performance

Critical strengths
  1. The paper emphasizes MLLMs’ limitations in reference-centric spatial reasoning and proposes that these limitations arise from models’ tendency to select camera-centric shortcut answers, which makes it difficult for them to produce reference-centric answers. This unified diagnostic framework and its findings are useful for disentangling the remaining failures across VLM benchmarks.

  2. OrientSAM achieves a meaningful improvement on ViewSpatial P-Rel, reaching 85.04% accuracy and outperforming APC-vis by 12.47 percentage points. This provides evidence of the proposed method’s effectiveness on the person-view relative direction task.

Critical weaknesses
  1. The paper claims that the authors identify reference-object orientation as a key factor governing shortcut strength. However, variation in camera-centric answer selection across orientation intervals could also reflect difficulties in orientation recognition or reference-frame transformation. It is therefore unclear how this conclusion follows from the observation that the rate at which the camera-centric answer is selected varies across orientation intervals.

  2. The evidence for consistent improvement on the paper’s central objective is limited. For example, the substantial gain on ViewSpatial P-Rel is meaningful, but P-Obj performance is lower than APC-Vis. The improvement on Spatial-MM Non-Cam is also modest at 0.74 percentage points, considering that the evaluation consists of 134 non-camera-view pairs.

  3. \citet is used throughout where \citep is required, making it unclear whether a citation is part of the sentence or a parenthetical reference.

Questions for authors

In Figure 2C, how can camera-centric and reference-centric left/right answers differ when the reference object is back-facing, where the two answers would generally be expected to be the same? Could you provide the number of samples in each angle interval, actual examples, and the data used to produce the plot?

Reviewer_sdToOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates camera-centric shortcut behavior in multimodal spatial reasoning, where MLLMs rely on image-plane cues instead of reasoning from a reference object's perspective. The authors propose OrientSAM, which incorporates explicit object orientation through Fourier-encoded orientation tokens and multimodal spatial alignment, together with a two-stage curriculum for orientation perception and reference-centric reasoning. Experiments on Spatial-MM, ViewSpatial, and 3DSRBench demonstrate improvements on non-camera-centric and orientation-sensitive spatial reasoning tasks.

Critical strengths
  1. The idea of camera-centric shortcuts as a failure mode in perspective-aware spatial reasoning is well motivated. The analysis provides a useful perspective for understanding this behavior.
  2. The paper is clearly written and the overall pipeline is straightforward. The motivation and methodology are clearly presented.
Critical weaknesses
  1. Although the paper compares OrientSAM with SpatialRGPT and APC-Vis, these comparisons primarily involve independently developed systems rather than controlled evaluations of different approaches under matched training conditions. Additional comparisons with existing perspective-aware spatial reasoning methods under a shared backbone and comparable training settings would better establish the methodological advantages of OrientSAM.
  2. Table 6 compares textual orientation prompting using SpatialRGPT against OrientSAM with a different backbone, making it difficult to isolate the benefits of the proposed embedding-based orientation injection. A controlled comparison using the same backbone, training data, and training settings would better establish the effectiveness of the proposed representation.
  3. The proposed framework already obtains object positions, depth, and orientation from external expert models. However, the paper does not compare OrientSAM against a geometry-based approach that explicitly performs reference-frame transformation using the same estimated spatial information. Such a comparison would help clarify the additional contribution of learned MLLM reasoning beyond the geometric information provided by the external experts.
Questions for authors
  1. Could the authors provide a controlled comparison between text-based orientation prompting and the proposed embedding-based orientation injection using the same backbone and training settings?
  2. Could the authors include a geometry-based baseline that directly computes reference-centric spatial relations using the estimated object positions, depth, and orientation? This would help clarify the additional benefits of learned MLLM reasoning. Furthermore, could the authors consider including other relevant spatial reasoning methods to provide a more comprehensive comparison and better establish the advantages of the proposed approach?
Reviewer_pKSKOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies camera-centric shortcuts in multimodal spatial reasoning: models often follow image-plane relations instead of reasoning from a reference object’s viewpoint. The proposed OrientSAM augments Qwen2.5-VL-7B with region-level RGB, depth, and Fourier-encoded orientation tokens, trained through an orientation-perception-to-spatial-reasoning curriculum. Experiments on three benchmarks show improvements concentrated on non-camera-view and orientation-sensitive tasks.

Critical strengths
  1. Clear and useful problem formulation. Separating consistent and conflicting reference-frame answers exposes failures that aggregate accuracy can hide.
  2. Well-motivated design. Explicit orientation provides a natural geometric cue for reference-frame transformation, while periodic encoding and staged training support its integration.
  3. Strong gains on targeted tasks. Improvements appear across multiple benchmarks, with ablations supporting the usefulness of orientation information and curriculum learning.
  4. Broad supporting analyses. Orientation shuffling, same-data fine-tuning, perturbation tests, and data deduplication help address alternative explanations.
Critical weaknesses
  1. Incomplete gain attribution: Missing same-backbone textual and direct geometric baselines limit evidence for the proposed representation’s advantage.
  2. Limited mechanistic evidence: Camera agreement and accuracy are mathematically dependent on binary conflict samples, weakening claims about shortcut mechanisms.
  3. Uncertain statistical reliability: Some leading margins are small, with only one training run and no confidence intervals.
Questions for authors

Do the gains from orientation-aware representations and curriculum training persist with a stronger backbone, such as Qwen3.5? This would help clarify whether OrientSAM remains effective as the backbone’s spatial reasoning capabilities improve.

Reviewer_Lu9LOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper identifies camera-centric shortcut behavior as an orientation-dependent failure mode in perspective-aware spatial reasoning, particularly when camera-centric and reference-centric answers conflict. To address this issue, the authors propose OrientSAM, an orientation-aware spatial alignment framework for multimodal models. OrientSAM injects explicit orientation information into multimodal representations through orientation-aware tokens and Fourier-based angle encoding, and further adopts a curriculum learning strategy to progressively improve perspective-aware reasoning. In addition, the authors build a spatial data construction pipeline to generate orientation-aware spatial supervision from large-scale images.

Critical strengths
  • The diagnostic is well targeted. Splitting consistent from conflict samples isolates the cases where the image-plane answer and the reference-frame answer disagree, which standard accuracy cannot reveal. Grouping by reference orientation turns a generic claim about egocentric bias into a concrete, measurable dependence.

  • The validation is unusually complete for a targeted method. Component ablations, a shuffled-orientation counterfactual, robustness sweeps over upstream estimator errors, a de-duplication check, an end-to-end efficiency comparison, and transfer to external benchmarks together make the behavioral claim credible. The appendix is candid about what each control does and does not show.

Critical weaknesses
  • The central claim is stronger than the evidence. On the primary target subset the gain over the strongest spatial baseline is under one point from a single training run, and the paper's own counterfactual leaves a large share of the improvement explained by depth, region tokens, and extra instruction data rather than by orientation. This undercuts the thesis that orientation is the missing factor.

  • The diagnostic establishes a behavioral pattern, not a mechanism. Because camera agreement and reference accuracy are complementary on binary conflict samples, citing reduced camera agreement as independent evidence for allocentric reasoning is circular. The abstract and the main results section should be reframed as accuracy gains on conflict subsets.

Questions for authors
  • Can you run a matched ablation with the same data, region tokens, and depth inputs but no orientation branch, to quantify how much of the gain comes from orientation alone?

  • The height subset is far below general API models. Is this a deliberate specialization trade-off or forgetting, and does it recover when more general spatial data are mixed in?

  • Would the conclusions change if the camera-centric answer accounted for depth and 3D layout rather than horizontal pixel position alone?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.