acceptodds
Under review as a conference paper at ICLR 2027

PanoArena: Zero-Shot Benchmark for Panoramic Understanding

Abstract

Existing evaluations of panoramic understanding remain largely image-centric and fail to systematically assess panorama-specific properties, particularly view-dependent spatial reasoning under the complete 360 field of view (FoV). To bridge this gap, we introduce PanoArena, a comprehensive benchmark with 4,631 samples for evaluating general-purpose panoramic understanding across both images and videos. PanoArena adopts a unified question-answering format with more than 10 sub-tasks spanning semantic and spatial understanding, where questions are further categorized into bearing-dependent and bearing-invariant settings based on whether the observer's view direction is required for reasoning. We systematically benchmark existing multimodal large language models (MLLMs) across different panoramic representations under zero-shot settings. We further propose PanoCompass, a training-free method, enabling a cleaner assessment of intrinsic panoramic reasoning rather than task-specific memorization. Extensive experiments reveal substantial limitations of current MLLMs, particularly in bearing-dependent spatial reasoning, providing new insights and directions for general-purpose panoramic understanding. The benchmark data and relevant code will be made available to the public.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
panorama, MLLM, benchmark, zero-shot

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_4tm8OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces PanoArena, a zero-shot benchmark for panoramic image and video understanding. It contains 4,631 questions across 11 static and dynamic sub-tasks, with each question labeled by whether answering it requires the camera’s reference frame. The authors compare panoramic input representations, model sizes, and thinking modes. They also introduce PanoCompass, a training-free baseline that uses projection, temporal cropping, and direction calculation tools. The results show persistent difficulties with direction and motion understanding, while tool assistance helps mainly on certain directional and local-perception tasks.

Critical strengths
  1. The benchmark covers both images and videos, including distance trends, motion direction, and target relocalization alongside recognition, OCR, and counting (Section 3.3). These video tasks require information across time and test abilities that single-image questions cannot capture.

  2. Camera-bearing dependence is labeled per question rather than per task (Section 3.2). The yaw experiments also distinguish consistent answers from answers that are correct at all four angles (Section 5.4). This is useful because similar average accuracy across rotations can hide changes in the answers to individual questions.

  3. PanoCompass uses an existing Qwen3.5-4B model without additional training. Table 2 shows egocentric direction accuracy increasing from 25.6% to 78.6%, while layout accuracy drops from 63.7% to 55.5%. Reporting both makes it easier to see where the tools help and where they still fall short.

Critical weaknesses
  1. Table 2 includes Qwen3.5 and several other models, but no Qwen3.8 or GPT-family model. Gemini is the only commercial model tested. I would like to see these models included to check whether the reported problems with direction and motion understanding still hold for newer models.

  2. In Appendix A, Table 5, reviewer agreement is 64.7% for Event Recognition and 71.8% for Motion Direction. These records include discarded samples and repeated reviews, so they do not tell us how reliable the final questions and answers are. The authors describe several rounds of revision and review, but a separate check of the final dataset would help distinguish model mistakes from questions that are still ambiguous.

  3. COL gives the model both the ERP and labeled cubemap views, while PanoCompass can request additional views, crop videos, and calculate directions with tools (Section 4). These settings give the model different observation and computation budgets, and Appendix B does not list the concrete visual-token budgets or inference costs. I would like to see comparisons at the same budget, along with results that remove each tool, to understand where the gains come from.

Questions for authors

See the weakness.

Reviewer_7GYZOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces PanoArena, a zero-shot benchmark comprising 4,631 QA pairs to evaluate multimodal large language models (MLLMs) on panoramic image and video understanding. The benchmark covers 11 static and dynamic sub-tasks and introduces a camera-bearing dependence annotation to measure spatial reasoning systematically. Additionally, the authors propose PanoCompass, a training-free agentic baseline that decomposes panoramic queries using standard projection and temporal tools. Validation is conducted by testing various MLLMs across different visual representations, which empirically shows that existing models experience measurable performance drops on bearing-dependent questions and motion reasoning tasks.

Critical strengths
  1. The introduction of the camera-bearing dependence taxonomy provides a highly effective diagnostic lens for spatial reasoning. By explicitly distinguishing whether a question requires the observer's view direction, the benchmark successfully isolates genuine 360-degree spatial understanding from pure semantic recognition. This design choice exposes a universal performance drop on bearing-dependent tasks across current MLLMs, making it an invaluable tool for future model development.

  2. The comprehensive coverage of both static and dynamic panoramic understanding addresses a significant gap in current evaluations. While existing benchmarks are largely image-centric, this work unifies 6 static and 5 dynamic sub-tasks to evaluate complex motion reasoning over time. Tasks like motion direction and target relocalization uniquely challenge models to track moving entities across a continuous spherical canvas, offering a much more holistic evaluation of an agent's panoramic perception.

Critical weaknesses
  1. The construction of dynamic reasoning tasks (such as Distance Trend and Motion Direction) heavily relies on automated geometric proposals generated by SAM3 and a depth estimator (DAP). While human reviewers verify these proposals, verifying accurate 3D distance trends or precise motion angles purely from distorted 2D panoramic video playback is notoriously difficult for humans. The inherent noise in the foundational depth models could introduce systematic biases into the ground truth of the dynamic subset, which may not be fully mitigated by human screening alone. This slightly undermines the reliability of the complex motion reasoning evaluation.

  2. The proposed PanoCompass baseline, while conceptually interesting as a training-free agentic approach, feels somewhat preliminary and loosely integrated with the core benchmark contribution. The method relies on a specific tool-calling interface (e.g., cropping, geometric calculation) and is only evaluated on the Qwen3.5-4B backbone. Furthermore, its performance gains are highly inconsistent—while improving egocentric direction, it noticeably degrades performance on global perception tasks like Layout (dropping from 63.7% to 55.5%). A deeper analysis of why the agentic decomposition fails on global reasoning would make this component much stronger and more convincing.

  3. The paper benchmarks models across different visual inputs (ERP, CMP, COL) and notes highly variable impacts; for instance, the COL representation helps some models but significantly hurts others on specific tasks (e.g., decreasing motion direction accuracy for Qwen-3.5-4B). However, the paper lacks an in-depth analysis of why spatial transformations like CMP or COL disrupt motion reasoning or how temporal consistency is degraded across these discrete views. Providing more insights into these failure modes would bridge the gap between empirical observation and actionable model design.

Questions for authors
  1. Given the heavy reliance on DAP depth estimates and SAM3 for generating distance trend and motion direction proposals, could the authors clarify if any quantitative sanity checks were performed to ensure that the inherent noise in zero-shot depth estimation does not introduce systematic biases? Specifically, how confident can we be that human reviewers effectively caught subtle 3D spatial errors from distorted 2D panoramic video playback?

  2. PanoCompass shows a notable performance drop on global perception tasks like Layout (dropping from 63.7% to 55.5%) and is currently only evaluated on the Qwen3.5-4B backbone. Could the authors clarify whether this degradation is a fundamental limitation of the tool-based decomposition (e.g., losing holistic global context when switching to local projections) or simply a capacity limit of the small 4B backbone? Have there been any preliminary tests of PanoCompass on larger models (like the 27B variant) to isolate this issue?

  3. The evaluation results indicate that using the Collage (COL) representation sometimes degrades motion reasoning accuracy (e.g., motion direction for Qwen3.5-4B drops from 24.3% to 22.8%) compared to the ERP baseline. Could the authors share their intuition on why this occurs? Does the discrete nature of the cubemap faces disrupt the model's ability to maintain temporal object consistency across frames? A brief discussion on this phenomenon in the rebuttal would be highly appreciated to deepen the paper's analytical value.

Reviewer_LreeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces PanoArena, a benchmark for zero-shot panoramic understanding with 4,631 questions covering 990 images and 1,512 video clips. It includes six static and five dynamic subtasks and annotates whether each question depends on the camera’s bearing. The authors evaluate multimodal models using equirectangular, cubemap, and collage representations, and examine model scale, thinking mode, and prediction consistency under camera-yaw rotations. They also introduce PanoCompass, a training-free baseline combining perspective projection, temporal cropping, and geometric calculation. The results identify weaknesses in camera-relative and motion understanding, while PanoCompass improves selected tasks, particularly egocentric direction reasoning.

Critical strengths
  • A meaningful benchmark scope covering both static and dynamic panoramic understanding. PanoArena evaluates image and video understanding within a unified QA framework, including direction reasoning, temporal perception, and motion reasoning. The dynamic tasks broaden the scope of panoramic evaluation and provide a useful setting for examining capabilities that cannot be adequately assessed from individual images.
  • A useful diagnostic perspective on camera-bearing dependence. The question-level distinction between bearing-dependent and bearing-invariant questions, together with the yaw-consistency evaluation, provides information beyond aggregate accuracy. This design helps characterize the relationship between panoramic reference frames and model predictions. The experiments also provide encouraging evidence that tool assistance can substantially improve selected capabilities, particularly egocentric direction reasoning.
Critical weaknesses
  • The comparisons of input representations, model sizes, and thinking modes in Table 2 do not isolate the contributions of PanoCompass’s projection, temporal cropping, and calculation tools. In particular, the large improvement in egocentric direction reasoning could plausibly be driven primarily by explicit geometric calculation after target localization. A localization-plus-calculation baseline and targeted component-removal experiments would help establish whether the complete framework provides additional value. Reporting observation and model-call budgets would also help distinguish the contribution of the framework from that of additional inference resources. This limits my confidence in the explanation of the method’s improvements.
  • PanoCompass combines intuitive operations into a practical training-free pipeline. The paper discusses existing training-free panoramic reasoning approaches, including Omni-CoT and Free360, but does not directly compare against them in Table 2. A comparison on supported static tasks, or a clear explanation of incompatibilities, would help establish the additional value of the proposed pipeline. I recognize that the benchmark is the primary contribution and do not require a new algorithm for a benchmark paper. However, the current methodological evidence provides limited additional support for acceptance beyond the benchmark itself.
Questions for authors
  • Can the authors provide targeted ablations separating projection, temporal cropping, and geometric calculation? In particular, how does the complete framework compare with a simple baseline that localizes the target on the ERP and deterministically converts its position into a direction?
  • What are the actual observation counts, model-call counts, and token budgets for PanoCompass relative to the ERP and COL baselines? Can the authors clarify whether its advantage persists under comparable inference budgets?
  • What additional capability does PanoCompass provide relative to Omni-CoT and Free360? A comparison on mutually supported static tasks would be useful; if such a comparison is not feasible, please explain the specific incompatibilities.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.