- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper addresses the limitations of conventional 2D bounding boxes in accurately representing pedestrian locations on the sphere in full-sphere panoramic pedestrian tracking. To address this issue, the authors introduce the PanoPed dataset and a lightweight spherical localization module called SEXTANT.
Critical strengths
-
The research problem is meaningful, and the proposed dataset is valuable. This paper addresses the localization representation problem in full-sphere panoramic pedestrian tracking, highlighting the limitations of conventional 2D bounding boxes in accurately representing visible pedestrian regions on the sphere. The proposed PanoPed dataset covers multiple camera platforms and provides rich synchronized geometric annotations, offering a useful resource for research on panoramic object tracking and spherical localization.
-
SEXTANT is lightweight and computationally efficient. With only approximately 35K trainable parameters, SEXTANT directly leverages existing detector features to refine spherical localization without introducing an additional image encoder or modifying the original identity association results.
-
The ablation studies are reasonably comprehensive. By keeping the detector and identity association results fixed, the authors compare ERP supervision, geometry-only refinement, capacity-matched geometric refinement, and the full method. These experiments help clarify the contributions of visual queries and spherical localization supervision.
Critical weaknesses
- The methodological novelty is relatively limited. SEXTANT is essentially a lightweight MLP that performs residual localization regression using existing detector features, while the main contributions appear to lie in the dataset construction. Furthermore, prior work such as 360VOTS (TPAMI 2025) has already investigated panoramic visual object tracking and segmentation, introduced the eBFoV representation, and provided dense pixel-level annotations. However, the paper does not sufficiently discuss its relationship to this closely related work.
- The real-world evaluation and generalization validation remain limited. The Sim-to-Real experiments primarily rely on segmentation masks automatically generated by SAM2.1 as evaluation targets, without independent human verification. In addition, although PanoPed contains a large amount of synthetic data collected from different camera platforms, the diversity of synthetic scenes is relatively limited. The real-world dataset contains only three densely annotated sequences, all captured using fixed cameras.
- The performance gains are relatively modest in real-world settings. Although SEXTANT achieves an improvement of approximately 1 HOTA point over the non-learned geometric baseline on real data, its additional gain over the learned geometric refinement module is only about 0.23–0.55 HOTA points. Furthermore, the Real-only LOSO experiments in the appendix show that training SEXTANT directly on limited real-world data actually degrades the performance of both MOTIP and HAT. These results suggest that the benefits of visual queries may depend on the amount of training data, feature quality, or domain distribution, and that the stability and generalizability of the method require further validation.
Questions for authors
-
Regarding methodological novelty, prior work such as 360VOTS has already investigated spherical object representations and pixel-level segmentation in panoramic videos. What are the fundamental differences between the proposed visible-support-based spherical localization targets and existing BFoV/eBFoV representations? Furthermore, what distinguishes SEXTANT from a conventional query-conditioned regression head, and why are its specific design choices necessary?
-
Since Support-auto relies on automatically generated segmentation masks as evaluation targets, have the authors considered manually annotating a subset of real-world data with accurate segmentation masks to validate the quality of the automatic masks and the resulting spherical localization targets? To what extent might segmentation errors affect the reported HOTA improvements?
-
In the Sim-to-Real experiments, SEXTANT achieves only modest additional gains over the capacity-matched geometric refinement module, while its performance even degrades under the Real-only LOSO setting. Could the authors further investigate the reasons behind these results and clarify under what conditions visual queries can consistently provide additional localization benefits beyond geometric information?
2 comments
The authors solve most of my questions, and I would like to raise my score to weak accept.
We apologize that our writing did not explain the methodological contribution clearly enough and caused confusion. We have rewritten it (Sec. 4.2 and the introduction, in blue). If the explanation below resolves your concerns, we would be very grateful if you would consider raising your score.
Q1a. How Sextant differs from a standard query-conditioned regression head. The detector already has such a head: its box head. It is trained to predict an ERP rectangle, so even a perfect box head leaves the conversion G to guess where the visible person lies inside the rectangle. Sextant keeps the same input but changes what is predicted and how:
Table 4 tests this design on PanoPed-S test:
A standard query head with the same input, network and budget gains nothing. Without the query, little remains. Only the query with the spherical target gives the full gain. On D-FINE-L, Sextant again beats the geometry head (+1.76 / +1.74 vs +0.16 / +0.15 HOTA, Sec. 5.5). The contribution is the finding that boxes lose spherical localization that queries still hold, and a minimal design that recovers it and transfers from simulation to real video.
Q1b. 360VOTS and BFoV/eBFoV. Thank you for the pointer. We added 360VOTS to Related Work and Table 1. BFoV and eBFoV describe the angular extent of an object, like a box on the sphere; our Box-fit layer follows eBFoV (App. D.4). Our target is the area-weighted center and spans of the visible pixels, so two people with the same eBFoV can have different targets. 360VOTS also tracks a single object. PanoPed tracks many people with persistent identities, adds depth and 3D states in PanoPed-S, and uses one target rule for synthetic and real data.
W2 / Q2. Automatic masks. The Support-auto masks are not produced by SAM2.1 on its own. Each mask is prompted by a human-reviewed box from our manual box annotation, which took about one month, and is computed in a local view centered on that box. The masks therefore have only small errors and are statistically meaningful. Our new human annotation on R-03 confirms this (General response, point 1): the annotator kept 93% of the SAM2.1 masks unchanged, masks drawn from scratch agree with SAM2.1 at a median IoU of 0.90, and against human targets Sextant still gains 1.08 and 0.87 HOTA over U. On scene diversity, PanoPed-S training varies 13 lighting conditions, 35 layouts and three platforms, and the real test is out of distribution by construction.
W3 / Q3. Gains over geometry, real-only LOSO, and when queries help. Gains over geometry. HOTA margin of Sextant over the capacity-matched geometry head (seeds 42 / 43):
The margin is positive in every setting and for both seeds. It is smaller on real video because the detector and the head are trained only on synthetic data and applied without fine-tuning.
Real-only LOSO. The drop under LOSO is caused by the training data, not by the method. Each LOSO fold trains on 2 real sequences: only 2 fixed viewpoints and the same 10 people. Its 75k to 91k observations are consecutive frames of these two views, so they are highly redundant and the 256-dimensional query input overfits to them. PanoPed-S instead offers tens of thousands of viewpoints. In simulation, without domain shift, the same head trained on 2 fixed sequences loses 2.54 and 2.84 HOTA, while 21,372 pairs from 44 sequences give +1.74 and +1.68 (General response, point 2). The LOSO detector is also trained on two sequences only, so its queries are weaker (MOTIP with U: 21.21 HOTA in LOSO vs 31.60 in sim-to-real). The paper therefore does not recommend LOSO as a standalone benchmark (Sec. 3.4) and reports it only for completeness.
When queries help. Queries help once the head sees many viewpoints, which PanoPed-S provides at low cost. The gain then holds on two detectors, all three synthetic splits, real video without fine-tuning, and human targets.