- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces PanoArena, a zero-shot benchmark for panoramic image and video understanding. It contains 4,631 questions across 11 static and dynamic sub-tasks, with each question labeled by whether answering it requires the camera’s reference frame. The authors compare panoramic input representations, model sizes, and thinking modes. They also introduce PanoCompass, a training-free baseline that uses projection, temporal cropping, and direction calculation tools. The results show persistent difficulties with direction and motion understanding, while tool assistance helps mainly on certain directional and local-perception tasks.
Critical strengths
-
The benchmark covers both images and videos, including distance trends, motion direction, and target relocalization alongside recognition, OCR, and counting (Section 3.3). These video tasks require information across time and test abilities that single-image questions cannot capture.
-
Camera-bearing dependence is labeled per question rather than per task (Section 3.2). The yaw experiments also distinguish consistent answers from answers that are correct at all four angles (Section 5.4). This is useful because similar average accuracy across rotations can hide changes in the answers to individual questions.
-
PanoCompass uses an existing Qwen3.5-4B model without additional training. Table 2 shows egocentric direction accuracy increasing from 25.6% to 78.6%, while layout accuracy drops from 63.7% to 55.5%. Reporting both makes it easier to see where the tools help and where they still fall short.
Critical weaknesses
-
Table 2 includes Qwen3.5 and several other models, but no Qwen3.8 or GPT-family model. Gemini is the only commercial model tested. I would like to see these models included to check whether the reported problems with direction and motion understanding still hold for newer models.
-
In Appendix A, Table 5, reviewer agreement is 64.7% for Event Recognition and 71.8% for Motion Direction. These records include discarded samples and repeated reviews, so they do not tell us how reliable the final questions and answers are. The authors describe several rounds of revision and review, but a separate check of the final dataset would help distinguish model mistakes from questions that are still ambiguous.
-
COL gives the model both the ERP and labeled cubemap views, while PanoCompass can request additional views, crop videos, and calculate directions with tools (Section 4). These settings give the model different observation and computation budgets, and Appendix B does not list the concrete visual-token budgets or inference costs. I would like to see comparisons at the same budget, along with results that remove each tool, to understand where the gains come from.
Questions for authors
See the weakness.