acceptodds
Under review as a conference paper at ICLR 2027

PanoAgent: Efficiently Understanding in 360° through Active Inspection

Abstract

Pretrained vision-language models (VLMs) exhibit strong perception and reasoning capabilities on perspective images, but panoramic projection distortions limit the use of these capabilities in panoramic understanding. Existing adaptation methods primarily rely on fine-tuning with panoramic question–answer supervision or predefined workflows that convert panoramas into planar views for the model. We introduce PanoAgent, which adapts VLMs to panoramic understanding by learning how to observe. It jointly learns to select question-relevant panoramic transformations and generate answers, allowing the model to actively select observations suited to its existing visual capabilities. Specifically, the model uses panorama-specific tools to select local planar views or update the global reference heading as needed, with returned evidence guiding subsequent observation and reasoning. Trajectory demonstrations connect panoramic concepts to observation actions, while reinforcement learning further refines the policy through outcome and geometric feedback. Experiments across multiple panoramic benchmarks show that PanoAgent achieves leading overall performance and offers a better balance among understanding accuracy, data efficiency, and inference efficiency than existing panoramic adaptation methods. The dataset and trained models will be released publicly.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Vision Language Model, Spatial Intelligence, Agent, Panorama Image Understanding

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_B8pDOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes PanoAgent, an active panoramic understanding framework that enables VLMs to select task-relevant observations through two tools: PlanarView for local perspective rendering and PanoRotate for reference-heading adjustment. The model is trained using trajectory-based supervised fine-tuning followed by GRPO with outcome and geometry-aware rewards. Experiments on multiple panoramic benchmarks demonstrate substantial improvements over base models and competitive panoramic adaptation methods.

Critical strengths
  1. Strong empirical performance. PanoAgent achieves substantial improvements over its corresponding base models across three panoramic benchmarks at both 4B and 9B scales. The results demonstrate the practical effectiveness of the framework.

  2. Effective training framework. The combination of trajectory supervision and geometry-aware reinforcement learning is well motivated. Ablation studies show consistent improvements from trajectory SFT and geometric rewards.

Critical weaknesses
  1. Insufficient isolation of the active observation contribution. The comparison between VQA SFT and trajectory SFT simultaneously changes supervision richness and access to additional visual observations. It remains unclear how much improvement specifically comes from adaptive observation rather than additional visual evidence. A compute or observation-budget-matched fixed multi-view baseline would strengthen the contribution.

  2. Limited evidence for efficiency claims. Training-data efficiency is measured primarily by the number of VQA examples without accounting for trajectory generation and teacher model costs. Furthermore, the PanoWorld inference time comparison uses a backbone proxy rather than the complete model, limiting the fairness of the efficiency analysis.

Questions for authors
  1. How does PanoAgent compare against a fixed multi-view or non-adaptive observation baseline under matched visual-token and inference budgets?

  2. In the Pixel versus Spherical interface comparison, could the approximate coordinate conversions introduce inconsistencies between training observations and actual inference-time views? How much do these inconsistencies affect performance?

  3. Could the authors provide more comprehensive training and inference cost comparisons, including trajectory synthesis overhead and actual end-to-end latency?

2 comments
AuthorsOpenReview ↗

Response to Reviewer B8pD

Thank you for recognizing the empirical results and for identifying the need to better separate observation adaptivity from additional evidence and computational cost.

1. Fixed observations and inference budgets. We evaluated ERP and fixed multi-view VQA-SFT baselines with increased visual budgets. Both remain less accurate than PanoAgent on 360Bench, ODI-Bench, and PanoSpace-Bench; General Response G3 (Table 2) reports the settings and results.

2. Pixel training–execution consistency. Both interfaces use the same source demonstrations. For supervised fine-tuning with the Pixel interface, we convert spherical local-view actions into ERP bounding boxes but retain the original observation images. At inference, the boxes are decoded into viewing directions and FoVs for rendering. This approximate conversion may produce views that differ from the training observations.

We compare the spherical coverage of the original training views and the views produced by the converted Pixel actions. The mean IoU is 0.9605. Differences are concentrated in views covering the poles, where ERP distortion limits how accurately rectangular boxes represent spherical views; other views match within numerical precision. Spherical actions specify viewing directions and FoVs directly, avoiding this conversion. Our comparison therefore evaluates the practical suitability of the interfaces, including their geometric constraints.

3. Training and inference costs. We use Qwen3.6-27B to generate and validate trajectory reasoning, producing approximately 40,000 trajectories in about 11 hours on eight H20 GPUs. For comparison, an SFT run of the 4B model using these data takes approximately 10 hours on the same GPU configuration. Since synthesis takes about as long as one SFT run and the generated data can be reused, we consider the upfront cost manageable.

For inference, the full PanoWorld-9B model, including its spherical module, takes 1.94 s per case. The updated measurement and the comparison with PanoAgent are reported in General Response G1 and G2 (Table 1).

Thanks to the authors for their response. I will maintain my original positive score.

Reviewer_KbEdOpenReview ↗
Rating
4
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PanoAgent, which selects visual observations by rendering selected panoramic regions as local planar views and adjusting the global reference heading. The model is first initialized using tool-use trajectories and then optimized through reinforcement learning with answer and geometric feedback.

Critical strengths
  • Using local planar views and panoramic rotation to address the challenges of ERP distortion is simple and intuitive.
  • Reprojecting panoramic regions into planar views makes effective use of VLMs’ pretrained visual capabilities and facilitates their transfer to panoramic understanding tasks.
  • The method achieves substantial improvements across multiple evaluation datasets.
Critical weaknesses
  • The ordering of the tables is confusing. For example, Table 6 appears immediately after Table 2.
  • PANOROTATE is restricted to the initial stage of the interaction, but the paper does not evaluate whether this restriction benefits the overall agent workflow.
  • During RL, the geometric reward takes the maximum score for each target across the trajectory and averages these scores. While this may discourage ineffective observation strategies to some extent, it does not account for when the highest-scoring observations occur within the trajectory.
Questions for authors

See weakness.

2 comments
AuthorsOpenReview ↗

Response to Reviewer KbEd

Thank you for the positive assessment and suggestions on the interaction protocol and reward design.

1. Restricting PanoRotate to precede PlanarView. Thank you for raising this question. Fixing the panoramic heading before local inspection keeps accumulated local observations in a consistent reference frame.

We compared our protocol, which permits PanoRotate only before the first PlanarView call, with a variant that permits PanoRotate at any stage. Our protocol retains only the latest ERP, while the variant without the timing restriction retains all ERPs.

On ODI-Bench, we evaluate scene-simulation cases where the model calls PanoRotate. On PanoSpace-Bench, we evaluate camera rotation and object-conditioned reorientation. These results support the effectiveness of our interaction protocol.

Table 1. Comparison of protocols with and without the PanoRotate timing restriction. Accuracy (%).

BenchmarkTaskWith restrictionWithout restrictionGain (pp)
ODI-BenchScene simulation58.952.6+6.3
PanoSpace-BenchCamera rotation77.272.0+5.2
PanoSpace-BenchObject-conditioned reorientation66.864.4+2.4

2. Timing of geometric feedback. Thank you for this suggestion. Our current geometric reward focuses on acquiring relevant evidence and improves accuracy across all three benchmarks, as shown in our reward ablation. We will explore incorporating observation timing into the reward and evaluate its effect on observation efficiency.

3. Table ordering. We will reorder the tables to follow their first citations and corresponding discussions in the revised manuscript.

Thanks to the author for replying to my comments. It has resolved some of my questions. I still maintain my original score.

Reviewer_GV93OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces PanoAgent, a vision-language agent for 360° panoramic understanding that learns question-dependent observation strategies to leverage pretrained VLMs’ capabilities on conventional images. The agent uses two geometric tools: PlanarView projects selected spherical regions into local planar views, while PanoRotate adjusts the panorama’s reference heading. It then uses the returned visual evidence to guide further observations or generate an answer. Training proceeds in two stages. First, observation trajectories are constructed from object geometry annotations and predefined task templates, verified and supplemented with reasoning text by a VLM, and used for supervised fine-tuning. Second, GRPO refines the policy using rewards for answer correctness, tool-call validity, target coverage, and heading alignment. Experiments with 4B and 9B models show improvements across multiple benchmarks for panoramic perception, spatial reasoning, and visual search, alongside lower training-data requirements and inference latency than some existing methods.

Critical strengths
  1. The paper studies an interesting and practically relevant problem. Panoramic understanding remains challenging despite the broad visual capabilities of current open-source VLMs. This is also consistent with my own informal observations when testing Qwen on panoramic images.
  2. The method is clearly presented, and the tools are well suited to the task. PlanarView supports detailed inspection of objects, while PanoRotate handles changes in the reference heading for directional reasoning.
  3. The experimental evaluation is thorough. Both the 4B and 9B models achieve substantial improvements over their corresponding base models across the three main benchmarks. Ablations on training strategies, tool use, action representations, and geometric rewards provide further support for the proposed design.
Critical weaknesses
  1. The analysis of inference cost remains incomplete. Although the paper reports average end-to-end latency, it lacks a comparison between active observation and direct answering using the same backbone in terms of latency, visual token count, and tool-call overhead. This makes it difficult to quantify the additional cost of the observation policy. Furthermore, the speed comparison with PanoWorld should use PanoAgent-9B under matched inference conditions.
  2. The accuracy of the two observation components is not evaluated separately. Tables 5 and 8 mainly report final answer accuracy, without directly assessing whether PlanarView covers the ground-truth target regions or whether PanoRotate aligns the panorama with the ground-truth reference heading. Reporting target coverage and heading error separately would clarify how accurately each component operates.
Questions for authors

Overall, I found the paper well written and the results convincing. The questions below are mainly intended for discussion. For the rebuttal, please prioritize the concerns raised in the Weaknesses section.

I tested GPT-6.1 Sol and GPT-6 Astra on a small set of 10 questions and obtained 100% accuracy. Have the authors conducted similar preliminary evaluations? I understand that broader testing requires additional resources, so this is not a request for new experiments. I am mainly curious whether panoramic understanding remains challenging for models such as Astra, although my small sample is clearly insufficient to draw a conclusion.

I found your discussion of seams in panoramic images interesting and would like to explore it further. When processing an image, a VLM divides it into patches and converts the two-dimensional image into a one-dimensional token sequence, typically in row-major order. This also separates adjacent image regions and, in a sense, introduces seams, yet VLMs have shown that two-dimensional positional encodings can address this issue. Since panoramic and conventional images are related through a nonlinear mapping, I would expect VLMs to learn this relationship without too much difficulty. PanoWorld already uses spherical spatial encoding and cross-attention to model panoramic geometry. Compared with directly enhancing the visual representation in this way, what motivated PanoAgent’s use of explicit view transformations?

Have the authors explored using 3D reconstruction to support spatial understanding? Since panoramas can be projected into perspective views, I wonder whether a model such as VGGT could provide useful geometry, as in single-image reconstruction. I would be interested in whether VLMs reason more effectively from panoramic images or from a suitable representation of the reconstructed 3D scene. If no such experiments have been conducted, please feel free to skip this question.

3 comments
AuthorsOpenReview ↗

Response to Reviewer GV93 (1/2)

Thank you for the careful review and constructive suggestions. Following your recommendation, we address inference cost and observation quality before the discussion questions.

1. Inference overhead and a matched PanoWorld comparison. We have added an inference-cost comparison between PanoAgent-9B and the full PanoWorld-9B model. At 4B, we compare PanoAgent-4B with our Qwen3.5-4B VQA-SFT model for direct answering. The results and tool-execution timings are reported in General Response G1 and G2 (Table 1).

2. PlanarView coverage and PanoRotate alignment. We evaluate PlanarView on 360Bench using target coverage, defined as the fraction of a target ROI’s spherical area covered by a rendered view. First-view coverage uses the first local observation, while Best-view coverage takes each target’s maximum coverage across all local observations. Both are averaged over targets within each case and then across cases with a local view. Coverage increases from 70.28% for the first view to 81.21% for the best views, showing that subsequent observations provide more complete coverage of the target regions.

We evaluate PanoRotate on the Rot. and Reori. tasks of PanoSpace-Bench, measuring heading error as the shortest circular distance between the resulting cumulative heading and the question’s target heading. PanoRotate matches the specified heading exactly on Rot. On Reori., the median error is 2.50°, with larger deviations in some cases raising the mean to 15.49°.

Table 1. PanoRotate heading alignment on PanoSpace-Bench.

TaskMean heading error (°)Median heading error (°)
Rot.0.000.00
Reori.15.492.50

Together, these results support the effectiveness of our learned policy in obtaining task-relevant panoramic observations.

3. Evaluations with GPT-6.1 Sol and GPT-6 Astra. We evaluated GPT-6.1 Sol and GPT-6 Astra on 360Bench, ODI-Bench, and PanoSpace-Bench using ERP inputs, with no tool calls.

Table 2. Overall and task-level accuracy on 360Bench (%).

ModelOverallFP-IRFP-ICPP-IRPP-ICSR-OsSR-OVDG
GPT-6.1 Sol61.4169.2043.3066.0050.5962.7369.4468.63
GPT-6 Astra69.2173.8646.3276.8760.6869.5578.4479.05

Table 3. Overall and task-level accuracy on ODI-Bench (%).

CategoryTaskGPT-6.1 SolGPT-6 Astra
AllOverall76.4078.84
GeneralObject attributes80.4981.23
GeneralHuman attributes79.9380.92
GeneralExistence88.5086.00
GeneralCounting59.3062.21
GeneralOCR86.7888.43
SpatialEgocentric view orientation81.4790.55
SpatialAllocentric view orientation65.8362.47
SpatialScene simulation65.0069.00
SpatialRelative direction82.0484.90
SpatialODI reasoning68.5071.00

Table 4. Overall and task-level accuracy on PanoSpace-Bench (%).

TaskGPT-6.1 SolGPT-6 Astra
Overall55.5458.06
Absolute direction64.8067.60
Relative direction47.2047.20
Camera rotation75.6080.80
Object-conditioned reorientation58.8064.80
Observer distance52.4052.80
Relative 3D position24.0028.00
ERP seam continuity66.0065.20

The two frontier models deliver SOTA-level overall performance on 360Bench and ODI-Bench, with strong results on basic perception tasks such as object and human attribute recognition. Their task-level results nevertheless show that spatial reasoning remains challenging across all three benchmarks. On PanoSpace-Bench, which focuses on panoramic spatial understanding, both models remain below 60% overall accuracy, compared with 65.54% for PanoAgent-4B and 71.14% for PanoAgent-9B. This contrast highlights PanoAgent’s effectiveness in panoramic spatial reasoning. We will include these additional evaluation results in the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer GV93 (2/2)

4. Motivation for explicit transformations. We agree that VLMs can, to some extent, learn the nonlinear geometric relationships between panoramic and conventional images. PanoWorld and our VQA SFT baseline both demonstrate the effectiveness of direct ERP adaptation. Our motivation is to make this adaptation more efficient by building on the model’s pretrained perception and reasoning capabilities.

We use explicit transformations to apply known geometry directly to the visual input, aiming to reduce the presentation differences that the model must accommodate through post-training. Local reprojection can jointly address seam-induced separation, distorted object appearance, and changes in local spatial layout. A unified observation policy selects question-relevant regions, fields of view, and reference headings, allowing the model to interpret panoramic evidence in more familiar visual forms.

Our data-scaling results support this motivation. On 360Bench, trajectory SFT trained on 25% of the source VQA pairs achieves 46.24% accuracy, surpassing VQA SFT trained on the full set (45.20%). This comparison shows that adaptation through learned observation can achieve stronger panoramic understanding with fewer training questions.

5. Using 3D reconstruction. Thank you for this suggestion. We agree that the broad scene coverage of panoramas makes them a promising input for recovering 3D geometry. In future work, we plan to explore how to represent the recovered geometry as input to VLMs and whether it can improve spatial understanding compared with using panoramic images directly.

I thank the authors for their thorough response. The clarifications and additional results address my main concerns. I remain positive about this paper and will keep my score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank all reviewers for their constructive feedback. To address the shared concerns about inference efficiency, we clarify our timing measurements, compare inference costs across model scales, and examine whether larger visual input budgets enable direct-answer baselines to match PanoAgent.

G1. Inference measurements

In the original paper, we approximated PanoWorld’s latency using its Qwen3.5-9B backbone served by vLLM. We have now adapted the full PanoWorld model, including its spherical module, to vLLM and measured an end-to-end latency of 1.94 s per case, compared with the original proxy estimate of 1.92 s. For PanoAgent’s tools, the average execution times are 43.13 ms per PlanarView call and 1.36 ms per PanoRotate call.

G2. Inference cost comparison

We compare PanoAgent with direct answering at the 4B and 9B scales, using our Qwen3.5-4B VQA-SFT model and PanoWorld-9B as the respective direct-answer baselines. Visual tokens count each initial input image and each new tool observation once.

Table 1. Inference cost comparison.

MethodVisual tokens/caseTool calls/caseEnd-to-end latency (s/case)
VQA SFT (4B)1,91901.12
PanoAgent-4B2,8551.461.86
PanoWorld-9B1,91901.94
PanoAgent-9B2,4231.212.43

PanoAgent-4B and PanoAgent-9B take 0.74 s and 0.49 s longer per case than their respective direct-answer baselines. PanoAgent-4B nevertheless achieves higher overall accuracy than PanoWorld-9B across all three benchmarks at comparable end-to-end latency (1.86 vs. 1.94 s per case).

G3. Performance with increased visual budgets

At the 4B scale, we further examine whether direct-answer baselines can match PanoAgent’s performance with larger visual input budgets. Alongside the ERP baseline, we include a fixed multi-view baseline to test whether predefined planar views can provide an effective alternative to adaptive observation. Its input consists of six question-independent 90° × 90° cubemap faces covering the full panorama, with direction labels and no additional ERP image.

Both baselines use VQA SFT with the same question–answer pairs and optimization settings. Under the paper’s default evaluation setting, the initial ERP resolution is capped at 2048 × 1024. For this comparison, we fine-tune the ERP baseline at 2560 × 1280 and the fixed multi-view baseline at 767 × 768 per view, then evaluate both at these resolutions on 360Bench, ODI-Bench, and PanoSpace-Bench.

Table 2. Fixed-observation baselines with increased visual budgets.

MethodVisual tokens/caseEnd-to-end latency (s/case)360Bench (%)ODI-Bench (%)PanoSpace-Bench (%)
ERP3,2001.8148.4065.9651.91
Fixed multi-view3,4561.6547.1961.5446.63
PanoAgent2,8551.8655.2772.3665.54

Even after fine-tuning with larger visual input budgets, the direct-answer baselines remain less accurate than PanoAgent on all three benchmarks.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.