- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper introduces EgoSR-Bench and its dataset OcPano: 3,577 multiple-choice questions (3 or 4 options) on multi-hop, observer-centred spatial reasoning from a single 360° equirectangular (ERP) panorama. The questions come from 119 panoramas of 15 photorealistic indoor scenes. They are generated from five reasoning templates (chained localization, direction–depth filtering, vertical–proximity, composite constraints, nested references) by executing spatial predicates over 3D object geometry, with hard distractors and LLM paraphrasing (English and Chinese).
About 30 MLLMs are evaluated under five input modes: raw ERP, or ERP plus candidate or main-object boxes or coordinates. The best mean accuracy is 38.28% (GPT-6-Astra), and candidate coordinates help most. SFT followed by GRPO raises accuracy in the coordinate modes to about 65–76% on an 857-question held-out set. A 12-person human study reports 34.67% accuracy.
Critical strengths
- A relevant gap. Composing several observer-centred relations from a single panorama, including ERP distortion and the 0°/360° seam, is not covered by perspective-image spatial benchmarks.
- A geometry-grounded label pipeline. Answers come from deterministic predicate execution over 3D object geometry. Each distractor violates at least one relation in the chain, and every item stores its logic chain, which in principle supports template- and relation-level analysis (Sec. 3.3, Fig. 3).
- Broad model coverage across proprietary and open models, and an input-mode design that probes how much explicit geometric cues help.
- Candid reporting of limitations in the supplementary material (S1, S4).
Critical weaknesses
- The panoramic format itself is not justified (Q1). The benchmark frames multi-hop egocentric reasoning around a single ERP panorama, but never tests whether the panorama is needed, or whether it is what makes the task hard. The authors acknowledge that ERP "distortion changes the apparent scale and shape of objects" and that the 0°/360° seam "can separate spatially adjacent objects onto opposite edges of the image" (p. 1). How the panoramas are produced (rendered from Replica meshes, or stitched from multiple views) is not described; stitched panoramas would add further artifacts such as duplicated or misaligned content at the seams. The same 360° content could be given as a set of perspective views from the same position, e.g., 6–8 images at 60°/45° yaw intervals, or a cubemap, each with a known heading. This is easy to render from the 3D scenes, avoids ERP distortion and seam splitting, and matches how most egocentric agents actually perceive (perspective cameras, often rotated). Without this control, low accuracy cannot be attributed to multi-hop spatial reasoning rather than to MLLMs' unfamiliarity with the ERP format, and the case for a panorama-specific benchmark is not made.
- Results are close to chance, for models and humans alike (Q2). With K ∈ {3, 4} options, chance is 25–33%, but no chance level is reported for the benchmark. Model means range from 21.77% to 38.28% (Table 2). Humans reach 34.67% against 29.00% chance, and only 29.17% against 30.00% chance on raw panoramas (Table S7). This suggests ambiguous questions or labels, e.g., whether "left/behind" is viewer- or object-relative, rather than hard reasoning.
Questions for authors
- How are the panoramas produced (rendered vs. stitched)? How do models perform on the same questions when the 360° view is given as multiple perspective images at fixed yaw intervals (e.g., 6–8 views at 60°/45°, or a cubemap) with known headings? Does the accuracy gap between ERP and multi-view input indicate that the difficulty comes from the format rather than from multi-hop reasoning?
- What is the per-question chance level, and how far above it is each model on Src? Why are human participants at chance on raw panoramas? Was the frame of reference for directional relations specified to participants and models?