acceptodds
Under review as a conference paper at ICLR 2027

EGOSR-BENCH: UNVEILING MULTI-HOP EGOCENTRIC SPATIAL REASONING IN PANORAMIC ENVIRONMENTS

Abstract

Understanding complex spatial relations from an egocentric perspective is fundamental for embodied agents to navigate, reason, and interact within real-world panoramic environments. While recent advances in three-dimensional (3D) scene understanding and visual question answering (VQA) have improved object recognition and grounding, structured multi-hop spatial reasoning remains significantly underexplored, especially under single-view constraints such as 360° panoramic observations. In this work, we introduce EgoSR-Bench, a benchmark for multi-hop spatial reasoning from panoramic egocentric input, challenging models to infer fine-grained spatial relations solely from a single red-green-blue (RGB) 360° observation. We propose a structured question-generation framework that leverages 3D scene geometry, object-level attributes, and large language model (LLM)-based linguistic refinement to construct logically grounded questions. Based on this framework, we build OcPano, the released dataset and manifest underlying EgoSR-Bench, with 3,577 structured instances from 15 scenes, multilingual variants, and dense spatial-semantic annotations. We further define evaluation protocols for diagnosing spatial reasoning performance and use post-training to study benchmark learnability.

Primary area:
datasets and benchmarks
Keywords:
Panoramic Vision, Multi-hop Egocentric Spatial Reasoning, Visual Question Answering, Embodied AI Benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_PfJ6OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces EgoSR-Bench and its dataset OcPano: 3,577 multiple-choice questions (3 or 4 options) on multi-hop, observer-centred spatial reasoning from a single 360° equirectangular (ERP) panorama. The questions come from 119 panoramas of 15 photorealistic indoor scenes. They are generated from five reasoning templates (chained localization, direction–depth filtering, vertical–proximity, composite constraints, nested references) by executing spatial predicates over 3D object geometry, with hard distractors and LLM paraphrasing (English and Chinese).

About 30 MLLMs are evaluated under five input modes: raw ERP, or ERP plus candidate or main-object boxes or coordinates. The best mean accuracy is 38.28% (GPT-6-Astra), and candidate coordinates help most. SFT followed by GRPO raises accuracy in the coordinate modes to about 65–76% on an 857-question held-out set. A 12-person human study reports 34.67% accuracy.

Critical strengths
  1. A relevant gap. Composing several observer-centred relations from a single panorama, including ERP distortion and the 0°/360° seam, is not covered by perspective-image spatial benchmarks.
  2. A geometry-grounded label pipeline. Answers come from deterministic predicate execution over 3D object geometry. Each distractor violates at least one relation in the chain, and every item stores its logic chain, which in principle supports template- and relation-level analysis (Sec. 3.3, Fig. 3).
  3. Broad model coverage across proprietary and open models, and an input-mode design that probes how much explicit geometric cues help.
  4. Candid reporting of limitations in the supplementary material (S1, S4).
Critical weaknesses
  1. The panoramic format itself is not justified (Q1). The benchmark frames multi-hop egocentric reasoning around a single ERP panorama, but never tests whether the panorama is needed, or whether it is what makes the task hard. The authors acknowledge that ERP "distortion changes the apparent scale and shape of objects" and that the 0°/360° seam "can separate spatially adjacent objects onto opposite edges of the image" (p. 1). How the panoramas are produced (rendered from Replica meshes, or stitched from multiple views) is not described; stitched panoramas would add further artifacts such as duplicated or misaligned content at the seams. The same 360° content could be given as a set of perspective views from the same position, e.g., 6–8 images at 60°/45° yaw intervals, or a cubemap, each with a known heading. This is easy to render from the 3D scenes, avoids ERP distortion and seam splitting, and matches how most egocentric agents actually perceive (perspective cameras, often rotated). Without this control, low accuracy cannot be attributed to multi-hop spatial reasoning rather than to MLLMs' unfamiliarity with the ERP format, and the case for a panorama-specific benchmark is not made.
  2. Results are close to chance, for models and humans alike (Q2). With K ∈ {3, 4} options, chance is 25–33%, but no chance level is reported for the benchmark. Model means range from 21.77% to 38.28% (Table 2). Humans reach 34.67% against 29.00% chance, and only 29.17% against 30.00% chance on raw panoramas (Table S7). This suggests ambiguous questions or labels, e.g., whether "left/behind" is viewer- or object-relative, rather than hard reasoning.
Questions for authors
  1. How are the panoramas produced (rendered vs. stitched)? How do models perform on the same questions when the 360° view is given as multiple perspective images at fixed yaw intervals (e.g., 6–8 views at 60°/45°, or a cubemap) with known headings? Does the accuracy gap between ERP and multi-view input indicate that the difficulty comes from the format rather than from multi-hop reasoning?
  2. What is the per-question chance level, and how far above it is each model on Src? Why are human participants at chance on raw panoramas? Was the frame of reference for directional relations specified to participants and models?
Reviewer_1bQjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes a benchmark for egocentric spatial reasoning from panoramic images. The paper tries to answer the question of whether current multimodal large language models reliably perform multi-hop spatial reasoning over a single ERP image and identify the correct target.

Critical strengths
  1. The paper is well written and easy to follow.
  2. Extensive experiments on the benchmark are presented for different model sizes.
Critical weaknesses
  1. The human level score is just 34% (vs. chance level score that is 29%). This raises a serious concern about the quality of the benchmark. If humans cannot answer such questions, it means either the questions are ambiguous, or the ground-truth answers are not correct or there is a structural problem in the benchmark design.

  2. Frontier models already achieve human-level scores or even marginally higher scores (e.g., gpt6 astra score is 38%). Also, the gap between frontier models and open-source small models like InternVL3.5-14B (BF16) is small (38% vs. 32%), which raises a major concern about how many samples of this benchmark can really be answered. Overall human, frontier models, and small model scores are all around the chance level.

  3. No blind test is applied. In a blind test, images should be removed, and the VLM will answer the question without seeing the image. This shows whether questions can be answered without looking at the image, based on the model's general knowledge or descriptions in the questions.

  4. Embodied agents (such as autonomous vehicles or autonomous robots) often have left/right/front/back observations. That is also closer to how humans see the surroundings. However, converting these images into a single panoramic image causes distortion in the image, which makes it harder, especially for spatial reasoning tasks.

  5. Post-training results are suspicious. The InternVL3.5-2 B model outperforms the InternVL3.5-14B model in 3 out of 5 modes. Qwen2-VL-7B outperforms the Qwen3-VL-8B in 2 modes. That raises the possibility that there is a problem in such modes. Prior work shows that in ego-centric multi-view observations, 14B internvl family models significantly outperform smaller-sized models [1].

[1] Spatial Reasoning with Vision-Language Models in Egocentric Multiview Scene

Questions for authors

See the weakness section

Reviewer_Dwo1OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

EgoSR-Bench studies target identification from composite spatial descriptions in a single panorama. OcPano contains 3,577 questions over 119 observations from 15 indoor scenes. Geometry grounds five reasoning templates, with linguistic refinement and distractor selection. The paper compares five visual/geometric input modes and studies learnability through SFT followed by GRPO on a separate 857-question test set.

Critical strengths
  1. A useful construction idea. Section 3.3 and Figure 4 combine geometric predicates, uniqueness filtering, and distractors that violate a relation. The template taxonomy usefully distinguishes chained references from jointly imposed constraints, giving the benchmark structure beyond a pooled accuracy score.

  2. An informative representation contrast. In Table 2, candidate coordinates improve over raw panoramas for all 32 configurations. Table 3 reports 73.98% with candidate coordinates versus 30.60% with raw panoramas for post-trained Qwen2-VL-2B. This motivates examining how geometry is represented to the model. It does not by itself identify perception or composition as the sole bottleneck, but the contrast is worth understanding.

Critical weaknesses
  1. The connection from executable geometry to the final question remains insufficiently established. Section 3.3 leaves the direction conventions and predicate thresholds unspecified. In Figure 3 (p.4), the question uses “left” and “adjacent to the sofa,” whereas its serialized record uses right and identifies the adjacency reference as window_4. An illustration error would not establish a dataset-wide defect, but this example cannot demonstrate the intended correspondence.

  2. The principal comparisons need a clearer link to their runs and denominators. Table 2 gives Gemini-2.5-Pro Src accuracy as 34.58%, while the pooled Src breakdown in Table S3 gives 6.68%; GPT-4.1-mini likewise has 35.56% versus 21.14%. Different runs or scoring rules could explain this, but the mapping is missing. For Table 3, Section 5.2 says invalid responses are not counted as correct, while Section S2 specifies that they reduce the valid denominator. Thus, a common 857-question test set need not mean a common scored population. Conditional accuracy can change with response validity as well as task performance. The separate post-training population is acknowledged on p.2; its provenance and relationship to OcPano still need explanation. I appreciate the disclosed limits on archived inputs and predictions, but those limits leave the central mode comparisons and learnability claims difficult to assess.

  3. Concern about the ambiguous reference. The “Zhengyuan Yang et al., VSI-Bench” entry (p.12, lines 601–602) could not be matched to a publication in the sources checked. VSI-Bench is introduced in Thinking in Space, Jihan Yang et al., already cited separately. Table 1 also describes it as a perspective-image benchmark without egocentric observations, whereas that paper evaluates egocentric video. The author lists for Pano-AVQA, Yun et al., ICCV 2021 and VQA-360°, Chou et al., WACV 2020 also differ from the publication records; the latter is incorrectly listed as a workshop paper.

Questions for authors
  1. How is question–label correspondence checked? Could you walk through one existing record from its geometric predicates to its final question and answer, making the reference frame explicit, and describe how broadly that correspondence has been validated?

  2. Which evaluation interpretation should readers use? Could you explain the run/scoring difference behind the Gemini-2.5-Pro Src discrepancy and use one existing Table 3 comparison to show the question population and treatment of invalid outputs?

  3. Which sources were intended? Could you identify or correct the separate VSI-Bench reference and reconcile the named entries and Table 1 with their original sources?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.