acceptodds
Under review as a conference paper at ICLR 2027

Think in 360°: Benchmarking and Bootstrapping Panoramic Video Reasoning

Abstract

A panoramic video captures all directions around the camera, but relevant evidence can appear in different directions and at different times. Understanding it requires models to connect observations across the viewing sphere and time. We introduce PanoVideoBench, the first manually annotated benchmark for comprehensive panoramic video understanding. Its 1,000 questions span five categories and 15 question types, with diagnostic annotations for target search and panoramic capabilities. Evaluation of 11 proprietary, open-source, and panoramic-specialized models identifies cross-view tracking and sphere-time evidence integration as their weakest capabilities. To strengthen these capabilities, we construct PanoVideoCoT, the first reasoning dataset grounded in real panoramic video, through sphere-time-grounded synthesis that converts verified panoramic observations and successful agentic explorations into 13,465 QA–CoT pairs. Starting from a general video MLLM, PanoVideoReasoner learns from this corpus through Panoramic CoT SFT and Evidence-Privileged On-Policy Self-Distillation, a panoramic instantiation of Vision-OPD's OPSD in which the teacher uses verified perspective views to guide student-generated trajectories. It reaches 55.7% on PanoVideoBench, improving the backbone by 12.5 points and the previous SOTA by 2.6. The gains concentrate on the same spatial, cross-view, and temporal capabilities exposed by the benchmark, showing that grounded sphere-time supervision can equip a general video MLLM with panoramic reasoning.

Primary area:
datasets and benchmarks
Keywords:
Panoramic Video Understanding, 360° Video, Multimodal Large Language Models, Spatiotemporal Reasoning CoT

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_rQkqOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates panoramic video understanding, where relevant visual evidence may be distributed across different viewing directions and temporal locations. The authors introduce PanoVideoBench, a manually annotated benchmark containing 1,000 multiple-choice questions across five semantic categories and 15 question types, with additional diagnostic annotations for evidence search and panoramic reasoning capabilities. Evaluations of 11 existing models reveal that cross-view tracking and sphere–time evidence integration remain challenging. To address these limitations, the authors construct PanoVideoCoT, a training dataset of 13,465 question-answer-reasoning pairs generated from real panoramic videos through sphere–time evidence construction, question synthesis, agentic exploration, and reasoning trace verification. They subsequently train PanoVideoReasoner using supervised Chain-of-Thought fine-tuning and Evidence-Privileged On-Policy Self-Distillation (EP-OPSD). The resulting model achieves 55.7% accuracy on PanoVideoBench, outperforming its Qwen3.5-9B backbone by 12.5 percentage points and the strongest reported baseline by 2.6 points. Overall, the paper contributes a structured evaluation framework and a practical training pipeline for panoramic video reasoning.

Critical strengths

The proposed PanoVideoBench extends beyond static panoramic perception to evaluate spatial relationships, action understanding, cross-view identity tracking, and temporal reasoning in real panoramic videos. A particular strength is the separation between semantic question categories, target-search settings, and panoramic reasoning capabilities. This design allows more informative diagnosis of model limitations than overall question-answering accuracy alone. The manually annotated evaluation set, independent answer verification, and direction-sensitive quality checks further support its value as a benchmarking contribution.

The proposed PanoVideoCoT pipeline associates visual observations across directions and timestamps, constructs questions from supporting evidence, and derives reasoning traces from successful visual exploration. This is a sensible approach to producing panoramic reasoning supervision while reducing dependence on fully manual annotation. The experiments show substantial improvements over the backbone, especially in spatial reasoning, motion tracking, and temporal reasoning. The ablation studies also provide useful evidence that CoT supervision and evidence-privileged teacher views contribute to the observed performance gains.

Critical weaknesses
  1. The evaluation does not sufficiently establish whether the model learns the intended sphere–time evidence-grounding capability.

The paper's central motivation is that panoramic video understanding requires locating, maintaining, and connecting observations across viewing directions and time. However, the main evaluation relies on multiple-choice answer accuracy. Although the benchmark includes diagnostic capability labels, these measure accuracy on particular question subsets rather than whether the model actually identifies and associates the correct supporting evidence.

This distinction is particularly important because the proposed training pipeline explicitly constructs evidence records containing timestamps, directions, and cross-view entity associations. These annotations could potentially support a more direct evaluation of evidence localization, identity continuity, and temporal association.

Related work such as CaST-Bench (CVPR 2026) has explored evaluating evidence-chain-grounded spatio-temporal reasoning. While the panoramic setting introduces additional challenges, the paper would benefit from discussing this distinction and providing stronger validation that the improvements reflect the intended evidence-association capability rather than improved multiple-choice video QA performance.

  1. The contribution and generalizability of panoramic-specific supervision are not sufficiently isolated from existing video reasoning and distillation approaches.

Several components build on established methods. For example, OmniCoT (ECCV 2026) explores panoramic reasoning benchmarks and CoT supervision, VideoEspresso (CVPR 2025) investigates grounded video reasoning data construction, and EP-OPSD explicitly adapts the existing Vision-OPD self-distillation formulation. The distinctive contribution therefore appears to lie primarily in extending evidence-grounded reasoning supervision to panoramic video and incorporating cross-view temporal associations.

While the reported improvements are encouraging, the experiments do not fully establish how much of the gain comes specifically from the proposed sphere–time representation and evidence association, as opposed to general CoT supervision and evidence-enhanced training. Moreover, the main evaluation is confined to PanoVideoBench, leaving the transferability of the learned capability to other panoramic-video or conventional video reasoning benchmarks unclear.

A more targeted ablation of the sphere–time evidence construction, or a cross-benchmark evaluation, would help establish the general significance of the proposed framework.

Questions for authors

Q1. How can the authors verify that the model correctly identifies and connects the relevant sphere–time evidence, rather than obtaining the correct multiple-choice answer through alternative cues?

Given that the training pipeline already maintains evidence timestamps, directions, and entity associations, could the authors evaluate evidence localization or association accuracy on a representative subset? Such an analysis would materially strengthen the central claim of improved panoramic reasoning.

Q2. How much of the improvement is attributable specifically to sphere–time evidence construction and association?

Table 5 establishes the benefits of CoT SFT and evidence-privileged distillation, but it would be helpful to isolate the value of the panoramic-specific evidence representation. For example, how does training with sphere–time-grounded CoTs compare with using conventional video CoTs or CoTs constructed without explicit cross-view entity association, under comparable training settings?

Q3. Does the learned reasoning capability generalize beyond the proposed benchmark?

Could the authors provide cross-dataset results on compatible panoramic-video benchmarks, or evaluate the trained model on an established conventional video reasoning benchmark? This would clarify whether the method develops transferable spatial-temporal reasoning capabilities rather than improving performance primarily on the proposed benchmark distribution.

Q4. How reliable are the automatically synthesized evidence associations and reasoning traces?

The pipeline uses model-based verification and successful agent rollouts to select training samples. Since correct final answers do not necessarily imply correct intermediate reasoning, is there any human assessment of the accuracy of the generated evidence chains and reasoning traces, particularly for cross-view identity tracking and temporal association? If available, reporting annotation agreement or error rates on a representative subset would help assess the reliability of the supervision.

Reviewer_mdXwOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces PanoVideoBench, a manually annotated benchmark containing 1,000 questions over 223 panoramic videos, with semantic categories and diagnostic capability labels. It also constructs PanoVideoCoT through evidence-grounded question generation and agentic exploration. PanoVideoReasoner combines CoT supervised fine-tuning with an evidence-privileged adaptation of Vision-OPD, reaching 55.7% accuracy, improving its backbone by 12.5 percentage points and the strongest evaluated baseline by 2.6 points.

Critical strengths
  1. The benchmark addresses a useful and relatively underexplored setting: finding and connecting evidence across viewing directions and time. Independent human review strengthens confidence in its answer labels.

  2. The training results show a substantial improvement over the backbone, with ablations demonstrating benefits from CoT supervision and evidence-selected teacher views.

Critical weaknesses

Limited methodological novelty and incomplete attribution. Section 4.4 explicitly retains Vision-OPD’s objective and training configuration. The main methodological contribution therefore rests on data construction, but Table 5 does not compare the proposed memory-and-exploration pipeline against direct CoT generation on the same QA pairs. Its necessity beyond ordinary synthetic supervision remains unclear.  All synthesis and certification roles use Qwen3.8-27B, without a reported independent human audit quantifying residual QA and CoT errors (Appendices B and E).  The reasoning diagnosis is not sufficiently controlled. Annotators inspect complete spherical videos, whereas models receive at most 128 frames from videos averaging 372 seconds (Sections 3.1, 3.3; Appendix C.2). No evidence-visibility audit establishes whether decisive events survive sampling and resizing. Single-frame and temporally shuffled controls are also absent, leaving uncertainty about which questions genuinely require temporal integration.  The representation comparison changes evidence coverage. The eight-view mosaic uses only equatorial views with 90° vertical fields of view, omitting the polar regions, and includes substantial overlap and an empty center tile (Appendix C.3). Consequently, its lower accuracy cannot be attributed specifically to difficulties reconstructing correspondence and spherical adjacency, as argued in Section 5.2.  The strength and generality of the gains remain uncertain. The 2.6-point lead corresponds to only 26 additional correct answers. Coverage is also uneven: TEM contains 74 questions and camera–object motion disambiguation only five (Figure 2). All reported training gains are evaluated on the proposed benchmark, leaving transfer beyond this evaluation untested.  Figure 1 presents a human baseline and a “40% Gap” without documenting the human evaluation protocol. The uniform and question-guided teacher-selection procedures in Table 5(b) are insufficiently specified to reproduce the comparison. Smaller writing issues include the incomplete phrase “stops in front of the framed” in Figure 1 and “Integrate evidences” in Figures 2–3.

Questions for authors

What proportion of benchmark questions remains answerable from the actual sampled and resized model input? How do single-frame and temporally shuffled baselines perform on the claimed temporal capabilities?

Reviewer_tXWWOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

PanoVideoBench contains 1,000 multiple-choice questions collected from 223 panoramic ERP videos in 360+x. The questions are created by annotators using a spherical video player, and then checked by two other annotators. Each question is assigned a semantic category (5 categories and 15 types), together with two additional labels. S0–S3 indicate how the relevant evidence should be found in the video, while C0–C4 describe the reasoning needed after locating it.

The paper also releases PanoVideoCoT, which contains around 13.5K QA-CoT pairs from 2,063 videos in 360-1M. The data is generated using a Qwen3.8-27B-based pipeline. It first builds an entity-level evidence memory, then uses a critic to refine the questions. An exploring agent is responsible for finding valid reasoning paths, and its successful results are further converted into CoT data. With this dataset, the authors train Qwen3.5-9B using LoRA. The training consists of two stages, starting with CoT SFT and followed by OPSD. For EP-OPSD, the teacher is also given perspective crops of the verified evidence, providing additional visual information during distillation. The final model reaches 55.7, which is 12.5 points higher than the original backbone and 2.6 points above Gemini 3.7 Flash.

Critical strengths

1.I think this is the strongest part of the paper. Panoramic video QA is still not well studied, and most existing datasets focus on single panoramas or audio-related tasks (Table 1). The annotation process also looks reliable. Annotators watch the videos and check different directions before writing questions, followed by reviews from other annotators. Directional questions are checked separately as well. 2. I like the separation between evidence search and reasoning. I initially expected unconstrained-search questions to be more difficult, but Fig. 2 shows this is not always the case. Some require a lot of searching but only simple answers. Even with a given direction, spherical reasoning can still be necessary. Table 4b also shows that the orientation overlay helps SPA/MOT/TEM, but hurts APP. 3.Table 5 is relatively easy to follow. The backbone and QA data are kept unchanged, so the effect of each component is easier to understand. I also checked the Overall scores against the category results. One result I found unexpected is that uniformly sampled extra views actually reduce OPSD performance (50.9 vs. 51.8). I am not sure why adding more views leads to worse results, but this is worth discussing.

Critical weaknesses

1.The claim that current models are weakest at cross-view tracking and sphere-time integration isn't backed yet. Fig. 1 shows a "Human" bar, but I couldn't find any human evaluation protocol or a per-category human breakdown. Without that, low MOT/TEM scores may only mean these questions are harder for everyone. The 128-frame cap (about one frame every 3 s at the average length) probably hurts counting and tracking too. TEM has only 74 questions, so small swings like Qwen's 29.7 → 18.9 in Table 4a (8 questions) shouldn't be interpreted.

2.It's unclear where the training gains come from. All results are on the authors' own benchmark, which shares its taxonomy with the training data. There is no external or general-video evaluation. CoT vs answer-only also mixes supervision content with thinking at test time. EP-OPSD has no "OPSD without extra views" run. About 5K CoTs come from the guided stage and are written with the answer location already known. No human check of their correctness is reported.

Questions for authors
  1. Where does the human number in Fig. 1 come from? Please give the protocol and a per-category breakdown.
  2. How was "difficulty" measured when keeping 17 of the 171 majority-vote items? If a model was used, its score is biased.
  3. How was overlap between 360+x and 360-1M checked?
  4. What is the text-only accuracy on the final 1,000 questions?
  5. Can you give confidence intervals? The +2.6 over Gemini is within about 1.6 SE. 5 Was the Qwen3.5-9B baseline run in thinking mode with the same token budget?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.