acceptodds
Under review as a conference paper at ICLR 2027

When the Agent Turns: Correctness-Aware Multimodal Reasoning over Panoramic Yaw Orbits

Abstract

When an embodied agent turns, the scene remains unchanged, but the correct answer to an egocentric spatial question may change. We identify a failure in panoramic vision–language models (VLMs): stable view-averaged accuracy can coexist withstale-frame predictions that retain the source answer when the reference frame requires an update. To measure this failure, we introduce a correctness-aware yaw-orbit benchmark that preserves image content through cyclic shifts and distinguishes invariant questions from equivariant questions whose targets are regenerated from 3D geometry. Across five open VLMs, mean transformed accuracy on invariant questions remains within 3.25 points of clean accuracy, yet full-orbit accuracy is 24.79–35.12% and changed-label accuracy is 16.25–30.58%. We propose OrbitAlign, a post-training framework that incorporates the answer transformation law into supervision and optimization. Orbit Supervised Fine Tuning (SFT) learns from geometry-regenerated targets; orbit-based Group Relative Policy Optimization (GRPO) jointly optimizes paired headings; and a correctness-gated Frame-Canonical Orbit (FCO) reward couples predictions only when both are correct in their respective frames. Across three backbones, OrbitAlign improves held-out changed-label accuracy over PanoEnv GRPO by 32.05–42.17 percentage points. With 100 Reinforcement Learning (RL) steps following orbit SFT, the complete framework reduces H200 training GPU-hours by a factor of 4.1–5.2. Ablations examine the contributions of target regeneration, orbit grouping, and cross-view correctness coupling. Together, these results establish current-frame correctness as a criterion for both evaluating and training panoramic spatial reasoning models. The benchmark, code, and model checkpoints will be released.

Primary area:
applications to robotics, autonomy, planning
Keywords:
3D Spatial Reasoning, Visual Question Answering, Panoramic Imaging, Embodied Agents

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_FmGJOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether panoramic VLMs can correctly update their answers when the observer’s heading changes. It introduces a yaw-orbit benchmark that distinguishes questions with invariant answers from those requiring answers to be regenerated from scene geometry. The proposed OrbitAlign combines orbit SFT, orbit-grouped GRPO, and FCO, substantially improving answer correctness in the current reference frame across three backbone models.

Critical strengths
  • The evaluation distinguishes answer consistency from correctness under reference-frame changes. The changed-label and full-orbit metrics reveal errors that view-averaged accuracy alone cannot capture, providing a valuable diagnostic approach for panoramic spatial reasoning.
  • The complete training framework achieves substantial improvements across three backbone models, demonstrating its effectiveness.
Critical weaknesses
  • In Table 4, adding FCO to orbit-grouped GRPO increases changed-label accuracy from 51.24 to 53.17, but decreases E1 robust accuracy from 45.87 to 45.04.
  • The evaluation is limited to the selected relative-position questions and does not demonstrate the fine-tuned models’ generalization to more diverse inputs and broader panoramic question-answering tasks.
Questions for authors

See weekness

3 comments

We sincerely thank the reviewer for their constructive feedback and valuable suggestions. We address each concern below.

  • FCO's E1 trade-off.

    • We agree that this result shows a trade-off. FCO is targeted toward E2 answer updating: its paired-heading reward emphasizes correctness in the current frame when a yaw change can change the answer. In Table 4, the FCO run reduces stale E2 answers by 12.13 points and raises changed-label accuracy by 1.93 points relative to orbit-grouped GRPO. At the same time, E1 full-orbit accuracy falls by 0.83 points. We have revised the manuscript to report that E1 cost alongside the E2 gains. The result supports FCO’s intended role in E2, while showing that it does not improve every metric.
  • Broader panoramic QA.

    • We agree that the selected yaw questions alone cannot show broader generalization. We therefore evaluated OrbitAlign and its base model on PCSR-Bench [1], which contains independently authored panoramic questions across six tasks. OrbitAlign improves on five tasks, raising the six-task average from 19.13% to 24.98%. Relative direction is the exception; an answer audit found that OrbitAlign frequently gives numeric responses where a directional answer is expected. We also evaluated transfer to real indoor panoramas from S2D3D. These results show gains beyond our yaw benchmark, while making the remaining failure clear.

Table 2: Six-task PCSR-Bench comparison on Qwen2.5-VL-7B. All rows use the 7,689 released test questions, shared inference settings, and a reconstructed relaxed scorer; the exact official scorer is unavailable. Base is our evaluation of the untrained model.RDis: Relative Distance; RDir: Relative direction CDC: directional chains; ERR: egocentric rotation; PRA: perspective re-anchoring. Avg. is the unweighted six-task mean. Bold marks the best of these rows; values are percentages.

ModelCountingRDisRDirCDCERRPRAAvg.↑
Base45.1512.3111.4436.373.236.2719.13
PanoEnv GRPO45.1522.7212.1742.922.6918.1523.97
OrbitAlign (ours)48.1525.095.6843.046.7321.1824.98

[1] Y. Chen, W. Wong, J. Li, I. Patras, and X. Zheng. Beyond localization: A comprehensive benchmark of perspective-conditioned spatial reasoning in MLLMs from omnidirectional images. In Proceedings of the 34th ACM International Conference on Multimedia, MM ’26.

Thank you to the author for replying to my comments. It has resolved some of my questions. I still maintain my original score.

Thank you for your thoughtful feedback and reconsideration. We appreciate your time and are pleased that our rebuttal addressed your concerns.

Reviewer_usxWOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies egocentric spatial QA on panoramas under yaw shifts. It means that pixels are preserved by an integer column roll, but egocentric answers must change with the reference frame.

A benchmark built on PanoEnv separates invariant questions (E1, full eight-heading orbits) from equivariant ones (E2, targets regenerated from rotated 3D geometry). Five open VLMs keep view-averaged accuracy within ~3 points of clean, yet full-orbit accuracy is only 24.79–35.12% and changed-label accuracy 16.25–30.58%.

The proposed OrbitAlign (orbit SFT on regenerated targets + orbit-grouped GRPO with a correctness-gated FCO reward) improves changed-label accuracy by 32–42 points over PanoEnv GRPO at 4–5x lower GPU cost.

Critical strengths
  1. Careful evaluation design: pixel-exact rolls, zero-yaw regeneration checks, seam filtering with sensitivity analysis, and a label-determinism audit showing 67.2% of changed targets are not inferable from source label + yaw.
  2. Convincing diagnostic: stable marginal accuracy coexists with large orbit-level failure across five models shown in Table 2 and Table 6. The correctness-vs-consistency framing is useful.
  3. Comprehensive reporting: orbit-data control, matched 100-step FCO ablation, per-angle breakdowns, S2D3D transfer, and a limitations section that discloses the unmatched-compute comparison and single-run ablation.
Critical weaknesses
  1. The headline comparison Table 3 is not initialization-matched. The ablation Table 4 shows orbit SFT alone lifts changed-label accuracy from 20.39 to 46.56; orbit grouping adds ~3 points and FCO only +1.93, while E1 robust accuracy drops 0.83. Most gains come from SFT on regenerated targets. Therefore, the RL machinery that carries the paper's methodological novelty contributes marginally.
  2. Training and evaluation targets share the same pinned PanoEnv relation rule, so E2 measures consistency with that rule on held-out scenes rather than independent spatial understanding.
Questions for authors

Q1. How much of the E2 gain survives without prompt coordinates? A text-only upper bound or larger marker study would clarify whether the model learns visual re-anchoring or coordinate arithmetic.

Q2. Do results hold under an independently authored label convention (e.g., human-annotated relations on S2D3D)?

2 comments

We sincerely thank the reviewer for their constructive feedback and valuable suggestions. We address each concern below.

  • Initialization and component attribution:

    • We agree that orbit SFT accounts for most of the improvement, and we have clarified what each comparison shows. Table 3 compares complete training pipelines, so it does not isolate the effect of RL. In Table 4, orbit SFT with regenerated targets raises changed-label accuracy from 20.39% to 46.56%. Starting from the same SFT checkpoint, standard GRPO reaches 48.21%, orbit-grouped GRPO reaches 51.24%, and the FCO run reaches 53.17%. Compared with orbit-grouped GRPO, the FCO run lowers the stale-answer rate from 46.01% to 33.88%—a 12.13-percentage-point drop—while E1 robust accuracy falls by 0.83 points.
  • How much survives without prompt coordinates?

    • We tested the same held-out E2 questions under two controls. When numerical coordinates are removed and objects are marked on the panorama, OrbitAlign reaches 38.60% changed-label accuracy, compared with 32.48% for PanoEnv GRPO (Table 1). A 6.12-point gain over PanoEnv therefore survives without prompt coordinates. When the panorama is replaced by gray but coordinates remain, OrbitAlign scores 34.62%, showing that coordinates also provide useful information. Its standard-input score is 61.97%. These results show coordinate-free improvement, while the gap to standard input shows that the full result depends on information available only in the original image-and-prompt setting.

Table 1: Coordinate dependence of E2 answer updating (held-out test). All rows use the same 948 seam-clean E2 records (702 changed-label pairs). Standard: original panorama with bounding-box coordinates in the prompt. Coordinates only: the panorama is replaced by a uniform gray image; prompts and coordinates are unchanged. Marker: A/B/C markers are drawn on the panorama and all numerical coordinates are removed from the prompt. Changed: accuracy on changed-label pairs (↑). Stale: fraction of parseable changed-label pairs that repeat the clean prediction (↓). All values are percentages.

StandardCoordinates onlyMarker
ModelChanged↑Stale↓Changed↑Stale↓Changed↑Stale↓
Qwen2.5-VL-3B
Base25.7880.1726.6471.7931.0551.14
PanoEnv GRPO23.3660.5428.2169.0932.548.01
OrbitAlign (ours)61.9742.4534.6248.4338.6043.02
  • Q2. Do results hold under an independently authored label convention?
    • This is an important distinction. Our S2D3D evaluation uses new real indoor scenes, but retains the PanoEnv relation rule; it tests scene transfer, not a second label convention. We therefore added PCSR-Bench [1], whose questions and answers were authored independently. OrbitAlign improves on PanoEnv GRPO in five of its six tasks and raises the average from 23.97% to 24.98% (Table 2). The exception is relative direction, where our answer audit identifies a substantial format failure. These results support transfer to independently authored panoramic tasks

Table 2: Six-task PCSR-Bench comparison on Qwen2.5-VL-7B. All rows use the 7,689 released test questions, shared inference settings, and a reconstructed relaxed scorer; the exact official scorer is unavailable. Base is our evaluation of the untrained model. RDis: Relative Distance; RDir: Relative direction CDC: directional chains; ERR: egocentric rotation; PRA: perspective re-anchoring. Avg. is the unweighted six-task mean. Bold marks the best of these rows; values are percentages.

ModelCountingRDisRDirCDCERRPRAAvg.↑
Base45.1512.3111.4436.373.236.2719.13
PanoEnv GRPO45.1522.7212.1742.922.6918.1523.97
OrbitAlign (ours)48.1525.095.6843.046.7321.1824.98

[1] Y. Chen, W. Wong, J. Li, I. Patras, and X. Zheng. Beyond localization: A comprehensive benchmark of perspective-conditioned spatial reasoning in MLLMs from omnidirectional images. In Proceedings of the 34th ACM International Conference on Multimedia, MM ’26.

Thank you for the helpful responses and showing those experiments.

My main concerns are resolved, and I'd like to maintain my original score for weak acceptance.

Reviewer_kV2mOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies an important failure of panoramic VLMs: after the agent turns, the model still answers in the old view even though the scene is unchanged. A benchmark is design to evaluate the problem, including some new data generation designs and accuracy metric improvements. A post-training framework named OrbitAlign is proposed with new SFT data and a improved version of GRPO to specifically solve the problem.

Critical strengths
  • The benchmark is reasonable and well motivated. The rotation keeps the image content unchanged, and the accuracy is computed in the current view over all angles, so the new evaluation is valuable.

  • The paper also provides a solution for this setting. The SFT and RL designs are built on the benchmark, and the ablation shows the effect of each component.

Critical weaknesses
  • The base models considered in the paper are out-of-dated. It would be necessary to check if the problem is still important in more recent models. If updated general pretraining/post-training has already mitigate the problem, the value of studying the problem would be very minor.

  • How about the model performance on existing benchmarks? Showing evidence based on third-party benchmarks would be more convincing.

  • I think modifying GRPO on the specific tasks would not be a major contribution -- it just add more constraints from the problem to the standard GRPO while not changing its basic designs/efficiency/effectiveness, although many recent papers do so.

  • To develop a good benchmark, it is necessary to add widely used baseline methods to show the real progress of VLMs on this tasks. It should include frontier models like gpt-6/claude-5 to check if the problem has been solved by frontier models, and widely used models / open-sourced modes to show average/common performance.

  • It would be better to have a name for the benchmark so that future work can use it easily.

Questions for authors

Please refer to the weaknesses part. W1/2/4 would affect my assessment.

After rebuttal: The authors' feedback addressed some of my concerns, including the performance of more advanced baselines and the effectiveness of the method on existing benchmarks.

4 comments

We sincerely thank the reviewer for their constructive feedback and valuable suggestions. We address each concern below.

  • W1/W4: Are newer and frontier models still affected?
    • To test whether recent and frontier models have already solved this problem, we evaluated seven hosted models on the same yaw benchmark. The hosted models, including recent Qwen and DeepSeek models, GPT-6.1 Sol, and Claude Opus 5.5. The problem persists even in the strongest models: GPT-6.1 Sol scores 62.40% on clean E1 views but 52.07% on complete orbits; Claude Opus 5.5 scores 67.36% and 53.72%, respectively. Their accuracy when the E2 answer must change is 28.65% and 47.11%. Thus, newer models improve performance but have not solved reliable answer updating after a turn. Table 1 reports all seven models alongside the open-weight baselines.

Table 1: Hosted-model yaw diagnostics on frozen validation sets. E1 uses 242 seam-clean eight-heading orbits; E2 uses 491 seam-clean records, including 363 changed-label records. Null = Clean × Mean; Kept = Robust / Clean. Stale counts repeated clean predictions when the target changes. All values are percentages.

E1 — invariantE2 — equivariant
ModelClean↑Mean↑NullRobust↑Kept↑Flip↓Mean↑Changed↑Stale↓
Hosted API models *
Qwen3.5-27B †50.8353.070.6035.1269.1140.0824.0323.6931.34
Qwen3.8-27B †54.1354.070.7335.5465.6541.7425.4620.9441.79
Qwen3.8-Max †53.7252.660.6039.6773.8542.6828.9227.0056.14
DeepSeek-V4.1-Flash54.9657.141.0939.2671.4331.7638.2935.5436.76
GPT-4o57.8560.741.7745.8779.2928.0430.9627.5543.80
GPT-6.1 Sol62.4061.332.0452.0783.4416.7735.0328.6568.87
Claude Opus 5.567.3665.173.3653.7279.7520.8440.7347.1152.89

All models get the same PanoEnv prompt and benchmark records and are graded by the official PanoEnv scorer. Output limits are 512 tokens for Qwen and DeepSeek, 4096 for GPT, and 8192 for Claude; Claude images are downscaled to 1536×768. Unreadable answers count as wrong. †10–34% of these Qwen responses hit the 512-token limit before answering.

  • W2: Does OrbitAlign transfer to an existing benchmark?
    • We added PCSR-Bench[1] to test transfer beyond our benchmark’s questions and label rules. OrbitAlign improves on PanoEnv GRPO in five of the six tasks and raises the six-task average from 23.97% to 24.98% (See Table 2). The exception is relative direction. Our answer audit identifies a substantial format failure there. This supports transfer across several independently authored tasks, while the direction result remains a limitation.

Table 2: Six-task PCSR-Bench comparison on Qwen2.5-VL-7B. All rows use the 7,689 released test questions, shared inference settings, and a reconstructed relaxed scorer; the exact official scorer is unavailable. Base is our evaluation of the untrained model. RDis: Relative Distance; RDir: Relative direction CDC: directional chains; ERR: egocentric rotation; PRA: perspective re-anchoring. Avg. is the unweighted six-task mean. Bold marks the best of these rows; values are percentages.

ModelCountingRDisRDirCDCERRPRAAvg.↑
Base45.1512.3111.4436.373.236.2719.13
PanoEnv GRPO45.1522.7212.1742.922.6918.1523.97
OrbitAlign (ours)48.1525.095.6843.046.7321.1824.98
  • W3: Is the GRPO change a major contribution?

    • We agree that a task-specific change to GRPO alone would not be a major algorithmic advance. The main contribution is identifying and measuring answer failures after a change in heading, then supervising the correct answer in each frame. The policy stage then aligns optimization with the benchmark’s central requirement: answering correctly at both headings. It groups paired views and weights each response’s update by correctness in the other view. We present it as a focused component of the complete framework.
  • W5: Benchmark name.

    • The benchmark is now named YawOrbit-Bench; OrbitAlign denotes the training framework.

[1] Y. Chen, W. Wong, J. Li, I. Patras, and X. Zheng. Beyond localization: A comprehensive benchmark of perspective-conditioned spatial reasoning in MLLMs from omnidirectional images. In Proceedings of the 34th ACM International Conference on Multimedia, MM ’26.

Thanks for your detailed feedback and extra results. The new results addressed my concerns about the baselines and the effectiveness of the method. I have upgraded my rating.

Thank you for your thoughtful feedback and reconsideration. We appreciate your time and are pleased that our additional results addressed your concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.