acceptodds
Withdrawn from ICLR 2027

Toward Comprehensive 3D Grounding: Orientation Grounding through Vision-Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Grounding is a core capability of spatial vision-language models, yet most existing work focuses only on where a referred object is. Many 3D tasks also require knowing how it is oriented. Although existing 3D VLMs may predict oriented boxes, box pose does not explicitly capture object-centric orientation or symmetry-induced ambiguities. We introduce orientation grounding, a referring grounding task that predicts an object’s 6D orientation and axial symmetry from a language or box query in single-view or multi-view scenes. To support this task, we construct ReferOri, with 331K multi-view and 387K single-view orientation-grounding queries obtained through scalable reconstruction, consistency checking, and human verification. We further present OG-VLM, which adapts a 3D VLM with structured box/orientation outputs, sign and symmetry tokens, and geometry-aware auxiliary losses. Across single-view and multi-view benchmarks, OG-VLM substantially outperforms orientation-aware VLM baselines and surpasses object-level orientation foundation models on scene-level referring benchmarks, showing that explicit orientation grounding is a distinct and learnable capability beyond localization. Downstream results validate its benefit for orientation-related spatial reasoning.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Orientaion grounding, Spatial Reasoning, VLM

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_NnftOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The authors propose the task of orientation grounding, which predicts an object's 6D orientation and axial symmetry in the 3D grounding task. To study this task, the authors introduce ReferOri, a referring orientation-grounding dataset. Then the authors present OG-VLM to solve this task. The experiments demonstrate the consistent performance gain on various benchmarks.

Critical strengths
  1. The proposed ReferOri dataset provides orientation and symmetry annotations for both single-view and multi-views. The proposed data generation pipeline could be potentially used for other data domain.
Critical weaknesses
  1. The proposed task and methods have limited novelty. In terms of the task, predicting object orientation is not new in 3D grounding. As described in Appendix C1, several existing methods, like SpatialReasoner and VG-LLM, already predict object orientations using different representations. It is not clear why existing orientation representations are not sufficient, and the proposed formulation in Sec. 3.1 worth exploring. In terms of the method, special tokens and geometric losses in Sec. 3.3 are not purely novel. For example, Grounded 3D-LLM introduces special referent tokens for 3D grounding, while VGR [A] uses special spatial tokens and an auxiliary geometric loss with an MLP prediction head. The proposed method extends these designs to the new task, but the novelty appear limited.
  2. The quality of the annotations in ReferOri depends heavily on the reconstruction and orientation estimation pipeline (e.g., SAM3D). For single-view data, only 5 rounds of 100 samples are manually checked. To verify the effectiveness of the proposed pipeline, the authors should consider compare against some independently established ground truth.

[A] Wang, Jiacong, et al. "VGR: Visual grounded reasoning." ICLR 2026.

Questions for authors
  1. Based on weakness 1, how does the proposed object-centric orientation differ from existing orientation representations? Could the authors provide a comparison to demonstrate why the formulated representation is necessary?
  2. Based on weakness 1, could the authors further clarify the novelty of the model design in sec. 3.3, compared against the existing work?
  3. Based on weakness 2, could the authors provide some analysis on the annotation pipeline against some datasets with ground truth annotations? Also, how good is the annotation pipeline when handling the occluded objects or symmetric objects where the orientation is hard to predict?
  4. For the baselines, could the authors also finetune those models using the same ReferOri data for a fair comparison? That would help distinguish the improvements from additional supervision and the proposed architecture.
  5. Could the authors provide an inference time computation comparison against the baselines (e.g. Spatial Reasoner and VG-LLM in Tab. 1)? Also, the authors should consider do computation analysis on the annotation pipeline (e.g., how long does it take to generate orientation for single-view and multi-views?). This would benefit other research which may potentially apply the pipeline on other datasets.
Reviewer_HXFeOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies predicting the position, intrinsic orientation, and symmetry properties of specified objects in a scene given language descriptions or bounding boxes. The authors construct ReferOri, containing 331K multi-view queries and 387K single-view queries, and build upon GS-Reasoner by adding dedicated output tokens, lightweight prediction modules, and geometric training objectives to form OG-VLM. Experiments cover orientation prediction, 3D localization, and spatial QA transfer.

Critical strengths

First, the task and data construction have value. Object position alone does not determine intrinsic front/back/left/right. The paper formalizes orientation-aware referring, provides 387K single-view and 331K multi-view queries with comprehensive manual verification, and offers a concise training scheme: dedicated sign/symmetry tokens plus lightweight prediction modules on GS-Reasoner. Task definition, annotation pipeline, and training form a complete and reusable package.

Second, experiments have broad coverage and demonstrate progressive improvement with downstream benefits. Progressive ablations on the same base model show added designs further improve orientation scores. Without task-specific QA training, the 3DSRBench orientation subset improves from 30% to 40%; MSQA lenient/strict matching rises from 29.02/18.05 to 32.64/28.18. With symmetry adjustment off, ScanRefer within-15° accuracy remains 63.4%, confirming gains are not solely from relaxed scoring.

Critical weaknesses

First, the task’s practical significance and the correspondence between labels and the required directions still need strengthening. The paper has not analyzed where existing systems fail due to orientation errors versus wrong-object or reference-direction confusion, leaving the task’s necessity partially unsubstantiated. Labels derive from canonical axes that may not match human-perceived directions; per-category “front” rules and label–human angular differences are not provided. Existing checks (97.5% agreement, Ori-COCO 45° bins) do not verify full 3D angular precision, and symmetry-to-rotation mappings are not listed. This does not imply label errors, but limits interpretation of orientation scores.

Second, individual contributions of added designs and orientation supervision have not been sufficiently isolated. The full method improves over “Only data” by 3.0–7.4 pp (within 15°), but individual contributions of dedicated tokens, prediction modules, and geometric objectives are not separated. Downstream comparisons change data, procedure, localization supervision, and design simultaneously; Table 12 shows localization also improves, so 3DSRBench gains cannot be solely attributed to orientation supervision. Downstream benefits cover only the orientation subset and MSQA, not comprehensive spatial reasoning. “Only data” details remain incomplete.

Questions for authors
  1. how do annotations and scoring correspond to objects’ intrinsic directions? Please explain the rules for determining an object’s front, right, and up; the equivalent rotations corresponding to each symmetry label; and the correspondence between ARKitScenes box directions and objects’ front faces. Does the existing manual review include directional records that can be further quantified? Alternatively, could an independent angular verification be conducted on representative samples with clearly determinable directions, reporting the error distribution?

  2. which training components produce the orientation-related QA benefits? Please specify the output format, prediction modules, and training weights of “Only data,” as well as the configuration corresponding to downstream “Ours.” The most discriminative supplement would be to fix the base model, scenes, inputs, and training budget and compare localization supervision alone versus adding orientation supervision; downstream scores from existing progressive configurations could also help assess the role of the added designs.

  3. similar design ideas can be found in recent works such as 3DThinker and Loc3RVLM, could you further explain the new insights its approach brings to related methods on comprehensive spatial reasoning tasks?

Reviewer_RfnrOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper studies orientation grounding, where a model predicts a referred object’s 3D box, object-centric orientation, and symmetry from single-view or multi-view scene inputs. Compared with traditional 3D visual grounding, which focuses on locating the referred object, this task also asks which way the object faces. It introduces ReferOri, a dataset built from existing referring datasets using 3D reconstruction and human verification. It also proposes OG-VLM, which adapts GS-Reasoner with structured output tokens and auxiliary geometric losses. Experiments compare orientation accuracy against spatial VLMs and dedicated orientation models, examine the proposed components through ablations, and evaluate transfer to spatial reasoning on 3DSRBench and MSQA.

Critical strengths
  1. The paper identifies a gap in current 3D grounding research, which mainly evaluates object location rather than object-centric orientation. It provides an explicit task formulation and benchmark for studying this distinction. This is a clear contribution, even though the practical need for explicit orientation prediction requires stronger evidence.

  2. ReferOri provides large-scale orientation supervision for both single-view and multi-view referring tasks. Training on this data substantially improves orientation accuracy, and the model also improves on spatial reasoning benchmarks without benchmark-specific fine-tuning. These results support the dataset’s potential as a resource for studying orientation learning and its transfer to spatial reasoning.

Critical weaknesses
  1. The paper cites the wrong work for the SR3D training dataset in Appendix E.1. The cited paper, Zhang et al. (2025), studies reconstruction for transparent and specular object grasping. The referring dataset used here should instead be credited to ReferIt3D by Achlioptas et al. (2020).

  2. The evaluation does not establish whether OG-VLM improves localization and orientation grounding jointly over existing baselines. Table 1 compares orientation accuracy without requiring correct localization, while Table 6 reports joint localization–orientation accuracy only for OG-VLM. The localization results in Table 4 and Appendix Table 12 are limited to internal ablations. Consequently, it remains unclear whether the orientation gains preserve localization performance or translate into better joint grounding relative to competing methods. Reporting localization, orientation, and joint accuracy for all baselines under the same evaluation protocol would better support the claim of more comprehensive 3D grounding.

  3. The paper does not report a simple two-stage baseline that uses an existing grounding model to locate the target and then applies Orient Anything to the object-centered crop. It also does not clarify whether Orient Anything was fine-tuned on ReferOri. A stronger comparison would fine-tune the orientation model on the same training data and evaluate the complete pipeline using predicted target boxes. This would help show whether OG-VLM improves over combining existing methods.

Questions for authors
  1. Symmetry definition and evaluation. Appendix B.3 assigns symmetry labels by reflecting the object’s point cloud. However, the evaluation treats some rotations as equivalent. Reflection symmetry does not always imply rotation symmetry. For example, a chair can have left-right reflection symmetry, but rotating it 180° around its vertical axis swaps its front and back, which are different because of the backrest. Could the authors explain how the reflection labels determine which rotations are equivalent? Also, OG-VLM uses its own predicted symmetry labels to adjust its orientation error, while baselines without symmetry predictions receive no adjustment. Could an incorrect symmetry prediction make the orientation score higher? How would the results change if all methods used the same symmetry rules?

  2. Motivation for explicit orientation prediction. The introduction uses a chair as an example and argues that knowing its location does not tell us which direction to sit down from. However, knowing its orientation is still not enough to perform the action. A robot policy could learn to predict actions directly from visual input without explicitly predicting the chair’s orientation. Some interactions may also depend more on finding the relevant object part. For example, opening a drawer may require finding its handle rather than identifying its canonical front. Could the authors explain which tasks benefit from explicitly predicting object orientation, compared with directly predicting actions or locating the parts needed for interaction?

  3. Could the authors explain the inconsistency between Tables 1 and 5? For the full model on ScanRefer, Table 5 reports Acc@30 of 69.03% on symmetric objects and 63.80% on non-symmetric objects, while Table 1 reports 70.1% overall. If the two groups cover the same test set under the same evaluation rules, the overall accuracy cannot exceed both group accuracies. The only-data model shows a similar inconsistency. Were different test subsets or evaluation rules used?

  4. Could the authors explain why the results with depth in Table 4 differ from Table 1? On RefCOCO+, the full model achieves Acc@15/Acc@30 of 32.2%/62.1% in Table 1, but 29.0%/58.7% in Table 4. The full model also performs worse than the only-data model in Table 4, while Table 1 shows the opposite trend. Were different training settings, test subsets, or evaluation rules used?

  5. Appendix B.6 states that two evaluators independently give binary judgments on 200 images from each dataset. If agreement is the percentage of images on which they agree, the score should change in steps of 0.5 percentage points. However, Table 9 reports values such as 96.27%, 98.79%, and 98.68%. Could the authors explain how these scores were calculated and what sample counts were used?

  6. Appendix E.4 states that Gram–Schmidt orthogonalization is applied to the two predicted axes before computing the orientation loss. Since these axes are already orthogonal, the orthogonality loss in Eq. (20) should always be zero. Is this loss actually computed on the vectors before orthogonalization?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.