acceptodds
Under review as a conference paper at ICLR 2027

PanoPed: Beyond Bounding Boxes for Sim-to-Real Panoramic Pedestrian Tracking

Abstract

Full-sphere panoramic cameras let fixed monitoring systems and mobile robots track people in every direction, but a planar bounding box does not fully describe where a person is on the sphere. We introduce PanoPed, a sim-to-real benchmark for pedestrian tracking on the full sphere. PanoPed-S contains 108,000 frames from fixed, quadruped-mounted, and drone-mounted cameras, with synchronized masks, depth, camera poses, and 3D pedestrian states. PanoPed-R adds 28,002 real frames from fixed cameras, 16,247 of them densely annotated. We find that an ERP rectangle cannot uniquely determine the spherical center and angular extent of the visible person, while the detector's visual query still carries information about them. Inspired by the sextant's use of angular measurements to locate objects, we propose Sextant, a plug-and-play angular localization head with only about 0.035M parameters. It reuses a frozen detector, keeps track identities unchanged, and needs no extra image encoder. Sextant gives the best result in our PanoPed-S test comparison, raising the strongest baseline, MOTIP, from 47.30 to 49.49 HOTA, with gains on all eight test sequences. Without fine-tuning on real data, the same synthetic-trained heads improve MOTIP and HAT by 0.96-1.14 HOTA on real video, and both seeds improve every real sequence. HAT+Sextant scores best among the compared systems that add no localization image encoder.

Primary area:
datasets and benchmarks
Keywords:
panoramic vision, multi-object tracking, spherical localization, datasets and benchmarks, synthetic data, sim-to-real transfer

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_vxGoOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper addresses the limitations of conventional 2D bounding boxes in accurately representing pedestrian locations on the sphere in full-sphere panoramic pedestrian tracking. To address this issue, the authors introduce the PanoPed dataset and a lightweight spherical localization module called SEXTANT.

Critical strengths
  1. The research problem is meaningful, and the proposed dataset is valuable. This paper addresses the localization representation problem in full-sphere panoramic pedestrian tracking, highlighting the limitations of conventional 2D bounding boxes in accurately representing visible pedestrian regions on the sphere. The proposed PanoPed dataset covers multiple camera platforms and provides rich synchronized geometric annotations, offering a useful resource for research on panoramic object tracking and spherical localization.

  2. SEXTANT is lightweight and computationally efficient. With only approximately 35K trainable parameters, SEXTANT directly leverages existing detector features to refine spherical localization without introducing an additional image encoder or modifying the original identity association results.

  3. The ablation studies are reasonably comprehensive. By keeping the detector and identity association results fixed, the authors compare ERP supervision, geometry-only refinement, capacity-matched geometric refinement, and the full method. These experiments help clarify the contributions of visual queries and spherical localization supervision.

Critical weaknesses
  1. The methodological novelty is relatively limited. SEXTANT is essentially a lightweight MLP that performs residual localization regression using existing detector features, while the main contributions appear to lie in the dataset construction. Furthermore, prior work such as 360VOTS (TPAMI 2025) has already investigated panoramic visual object tracking and segmentation, introduced the eBFoV representation, and provided dense pixel-level annotations. However, the paper does not sufficiently discuss its relationship to this closely related work.
  2. The real-world evaluation and generalization validation remain limited. The Sim-to-Real experiments primarily rely on segmentation masks automatically generated by SAM2.1 as evaluation targets, without independent human verification. In addition, although PanoPed contains a large amount of synthetic data collected from different camera platforms, the diversity of synthetic scenes is relatively limited. The real-world dataset contains only three densely annotated sequences, all captured using fixed cameras.
  3. The performance gains are relatively modest in real-world settings. Although SEXTANT achieves an improvement of approximately 1 HOTA point over the non-learned geometric baseline on real data, its additional gain over the learned geometric refinement module is only about 0.23–0.55 HOTA points. Furthermore, the Real-only LOSO experiments in the appendix show that training SEXTANT directly on limited real-world data actually degrades the performance of both MOTIP and HAT. These results suggest that the benefits of visual queries may depend on the amount of training data, feature quality, or domain distribution, and that the stability and generalizability of the method require further validation.
Questions for authors
  1. Regarding methodological novelty, prior work such as 360VOTS has already investigated spherical object representations and pixel-level segmentation in panoramic videos. What are the fundamental differences between the proposed visible-support-based spherical localization targets and existing BFoV/eBFoV representations? Furthermore, what distinguishes SEXTANT from a conventional query-conditioned regression head, and why are its specific design choices necessary?

  2. Since Support-auto relies on automatically generated segmentation masks as evaluation targets, have the authors considered manually annotating a subset of real-world data with accurate segmentation masks to validate the quality of the automatic masks and the resulting spherical localization targets? To what extent might segmentation errors affect the reported HOTA improvements?

  3. In the Sim-to-Real experiments, SEXTANT achieves only modest additional gains over the capacity-matched geometric refinement module, while its performance even degrades under the Real-only LOSO setting. Could the authors further investigate the reasons behind these results and clarify under what conditions visual queries can consistently provide additional localization benefits beyond geometric information?

2 comments

We apologize that our writing did not explain the methodological contribution clearly enough and caused confusion. We have rewritten it (Sec. 4.2 and the introduction, in blue). If the explanation below resolves your concerns, we would be very grateful if you would consider raising your score.

Q1a. How Sextant differs from a standard query-conditioned regression head. The detector already has such a head: its box head. It is trained to predict an ERP rectangle, so even a perfect box head leaves the conversion G to guess where the visible person lies inside the rectangle. Sextant keeps the same input but changes what is predicted and how:

  1. Target. It predicts the visible support on the sphere instead of a rectangle, so two people who share a box can receive different regions.
  2. Parameterization. It predicts a residual from the box region in units of the box's angular size, so the correction is small, independent of distance, and valid at the seam and the poles.
  3. Attachment. Its zero-initialized output reproduces the box region before training. The head starts from the tracker's own output, learns only the difference, and leaves identities and scores unchanged.

Table 4 tests this design on PanoPed-S test:

ReadoutQueryTargetHOTA (42 / 43)
Geometric readoutnonone47.30
Standard query headyesERP box47.24 / 47.26
Geometry-only headnosphere47.61 / 47.50
Sextantyessphere49.49 / 49.47

A standard query head with the same input, network and budget gains nothing. Without the query, little remains. Only the query with the spherical target gives the full gain. On D-FINE-L, Sextant again beats the geometry head (+1.76 / +1.74 vs +0.16 / +0.15 HOTA, Sec. 5.5). The contribution is the finding that boxes lose spherical localization that queries still hold, and a minimal design that recovers it and transfers from simulation to real video.

Q1b. 360VOTS and BFoV/eBFoV. Thank you for the pointer. We added 360VOTS to Related Work and Table 1. BFoV and eBFoV describe the angular extent of an object, like a box on the sphere; our Box-fit layer follows eBFoV (App. D.4). Our target is the area-weighted center and spans of the visible pixels, so two people with the same eBFoV can have different targets. 360VOTS also tracks a single object. PanoPed tracks many people with persistent identities, adds depth and 3D states in PanoPed-S, and uses one target rule for synthetic and real data.

W2 / Q2. Automatic masks. The Support-auto masks are not produced by SAM2.1 on its own. Each mask is prompted by a human-reviewed box from our manual box annotation, which took about one month, and is computed in a local view centered on that box. The masks therefore have only small errors and are statistically meaningful. Our new human annotation on R-03 confirms this (General response, point 1): the annotator kept 93% of the SAM2.1 masks unchanged, masks drawn from scratch agree with SAM2.1 at a median IoU of 0.90, and against human targets Sextant still gains 1.08 and 0.87 HOTA over U. On scene diversity, PanoPed-S training varies 13 lighting conditions, 35 layouts and three platforms, and the real test is out of distribution by construction.

W3 / Q3. Gains over geometry, real-only LOSO, and when queries help. Gains over geometry. HOTA margin of Sextant over the capacity-matched geometry head (seeds 42 / 43):

SettingMargin
PanoPed-S test, MOTIP+1.97 / +1.83
PanoPed-S test, D-FINE-L+1.60 / +1.59
Real video, MOTIP+0.55 / +0.24
Real video, HAT+0.55 / +0.23
R-03, human targets, HAT+0.58 / +0.44

The margin is positive in every setting and for both seeds. It is smaller on real video because the detector and the head are trained only on synthetic data and applied without fine-tuning.

Real-only LOSO. The drop under LOSO is caused by the training data, not by the method. Each LOSO fold trains on 2 real sequences: only 2 fixed viewpoints and the same 10 people. Its 75k to 91k observations are consecutive frames of these two views, so they are highly redundant and the 256-dimensional query input overfits to them. PanoPed-S instead offers tens of thousands of viewpoints. In simulation, without domain shift, the same head trained on 2 fixed sequences loses 2.54 and 2.84 HOTA, while 21,372 pairs from 44 sequences give +1.74 and +1.68 (General response, point 2). The LOSO detector is also trained on two sequences only, so its queries are weaker (MOTIP with U: 21.21 HOTA in LOSO vs 31.60 in sim-to-real). The paper therefore does not recommend LOSO as a standalone benchmark (Sec. 3.4) and reports it only for completeness.

When queries help. Queries help once the head sees many viewpoints, which PanoPed-S provides at low cost. The gain then holds on two detectors, all three synthetic splits, real video without fine-tuning, and human targets.

The authors solve most of my questions, and I would like to raise my score to weak accept.

Reviewer_EuG8OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies pedestrian multi-object tracking with panoramic (360-degree) cameras. It introduces PanoPed, a sim-to-real benchmark, and SEXTANT, a lightweight localization head.

Critical strengths
  1. The benchmark fills an important gap and uses a consistent target definition.

PanoPed uses the same visible-support rule to define spherical targets in both synthetic and real data. It covers polar regions, includes multiple moving camera platforms, and provides synchronized masks, depth maps, and 3D states. Table 1 shows that existing datasets, such as JRDB-PanoTrack, QuadTrack, and 360VOT, do not offer all these properties together. The unified target definition enables direct evaluation of sim-to-real transfer. I consider this the main contribution of the benchmark.

  1. The experiments are carefully controlled and support the main findings.

The authors use three controlled comparisons to rule out alternative explanations: a query head trained with ERP-box supervision, a geometry-only head without query features, and a parameter-matched geometry head. Only the combination of visual queries and native spherical supervision achieves the full improvement (Table 4). The authors also document their evaluation timeline, including the decision to inspect V-audit only at the final stage. They report sequence-level bootstrap results, per-sequence performance, and qualitative examples selected using fixed rules. They also report negative results, including the performance drop when SEXTANT is trained only on real data.

I appreciate this level of experimental transparency. It is uncommon in similar work and is a major reason for my positive recommendation.

Critical weaknesses
  1. The practical significance of the improvements remains limited.

SEXTANT improves HOTA by approximately 2.2 points on synthetic data and 1 point on real data. The IDF1 improvement is only around 0.3 points. Since the method does not change identity assignments, all improvements come from changes in localization overlap. More importantly, Table 18 shows that directly applying SAM2.1 to predicted boxes achieves 38.07/38.24 HOTA on real data. These results are much higher than SEXTANT's 32.74/32.85. The authors quantify SAM's bias toward its own masks in Table 20 (approximately 0.094 IoU). However, SAM may still outperform SEXTANT even after accounting for this bias. The paper's evaluation target also partly defines the problem of information loss in ERP boxes. The authors do not show whether more accurate spherical localization improves downstream tasks, such as robot navigation or 3D localization.

  1. The real-world dataset is small, and the evaluation targets lack manual verification.

Only three real-world sequences have dense annotations. All sequences use a fixed camera and involve the same group of ten participants. The dataset does not include real footage from moving camera platforms. The main evaluation target, Support-auto, relies on masks generated automatically by SAM2.1. The authors explicitly state that they did not manually verify these masks. Therefore, the sim-to-real conclusions depend on automatically generated labels. When the authors train SEXTANT only on real data under the leave-one-sequence-out (LOSO) setting, HAT's HOTA decreases by 0.21–0.26 points. This result raises questions about how the method behaves under real-world feature distributions. The synthetic training and test splits also share the same scene family and the same set of 57 virtual human models. This limits the evaluation of out-of-distribution generalization.

Questions for authors
  1. Could the authors manually annotate masks for a random subset of Support-auto? Please report the agreement between SAM2.1 masks and human annotations. It would also be useful to compare SEXTANT and the SAM2.1 readout again on this subset.

2.After accounting for SAM's bias toward its own masks, how do the authors justify the practical value of SEXTANT compared with a lightweight segmentation-based readout? Could the authors compare accuracy and inference speed against a much smaller segmentation model, such as MobileSAM?

3.SEXTANT reduces performance under the real-data LOSO setting. Do the authors attribute this mainly to overfitting or to a shift in the query feature distribution? Does this result suggest that the method depends on the scale of synthetic training data?

2 comments

Thank you for the detailed review and for valuing the controls and the reporting.

W1 / Q2. Practical value compared with segmentation readouts. New experiment (Sec. 5.5; App. F.11, Table 24), sim-to-real with HAT, readout time on one RTX 5090 D:

ReadoutParametersms per frameHOTAIDF1
U (geometric)0031.7131.95
Sextant 42 / 430.035M0.6332.85 / 32.6832.28 / 32.28
MobileSAM10.13M696.935.0032.18
SAM2.1 Hiera-L224.45M789.938.2432.68

SAM2.1 and MobileSAM are more accurate, but they have about 6,500 and 290 times the parameters of Sextant and are about 1,250 and 1,100 times slower. A latency of 0.7 to 0.8 s per frame would be disastrous for a robot that must react to people in real time. Sextant adds 0.6 ms per frame and keeps the tracker at 11 fps. Part of their lead also comes from the SAM-made targets: against human masks drawn from scratch, SAM2.1 and MobileSAM lose 0.050 and 0.024 IoU, while Sextant does not (General response, point 1).

On the small IDF1 change: Sextant does not change identities by design, so IDF1 moves only through matching. Localization does improve. On PanoPed-S test, LocA rises from 82.54 to 84.67, and the HOTA gain grows at strict IoU thresholds (Fig. 4).

On the target defining the problem: the target is fixed by the visible pixels and built by the same rule in S and R. It does not depend on any method. A box-based target cannot measure where the visible person lies inside the box, and this is the error we study.

On downstream tasks: we did not evaluate navigation or 3D localization. The center direction is the bearing a robot uses to follow or avoid a person, and Sextant reduces its error from 0.524° to 0.412° on V-dev matches.

W2 / Q1. Real data and human masks. We annotated masks by hand on R-03 and compared Sextant and SAM2.1 again on this subset (General response, point 1). The box-prompted SAM2.1 masks are largely accurate, and against human targets Sextant still gains 1.08 and 0.87 HOTA over U. The sim-to-real test is out of distribution by construction: real scenes, real people and a real camera.

Q3. LOSO. Mainly overfitting. The drop is caused by the training data, not by the method: with two fixed viewpoints, the head overfits (General response, point 2). In simulation, two fixed training sequences already make Sextant worse (−2.54 and −2.84 HOTA), while fewer pairs from 44 sequences make it better (+1.74 and +1.68). The LOSO detector is also trained on only two real sequences, so its queries are weaker. On scale: the head depends on the number of viewpoints more than on the number of samples, and simulation provides many viewpoints at low cost.

The author has addressed my concerns, I will maintain my positive score!

Reviewer_ZSaFOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies pedestrian tracking in full-sphere panoramic videos and introduces the PanoPed dataset and a lightweight localization module called SEXTANT. Its central observation is that an ERP bounding box cannot uniquely determine the spherical center and angular extent of a pedestrian’s visible pixels, while the detector’s visual query may retain this information. SEXTANT uses the query, an initial spherical region, and the detection confidence to predict localization corrections, while leaving the original identities and confidence scores unchanged. The method improves MOTIP from 47.30 to 49.49 HOTA on PanoPed-S and achieves gains of approximately 0.96–1.14 HOTA on PanoPed-R without fine-tuning on real data.

Critical strengths
  1. The problem is meaningful and the motivating analysis is concrete. An ERP box does not determine the spherical center or angular extent of the visible person, and two people in different poses sharing one box receive the same region. Box-based evaluation hides this error, which motivates a sphere-native representation.
  2. SEXTANT is simple and parameter-efficient. Since the detector and identity outputs remain frozen, the performance gains can be attributed relatively clearly to improved localization.
  3. The ablation studies show that neither visual queries alone nor geometry-only calibration reproduces the full improvement, providing reasonable support for the paper’s central claim.
Critical weaknesses
  1. The main weakness is that the real-world experiments remain limited in their ability to support broad generalization claims. PanoPed-R contains only three densely annotated sequences and ten participants, and all recordings were captured using fixed cameras. This is insufficient to establish generalization to more diverse real-world conditions.
  2. The primary real-world evaluation targets are generated automatically by SAM2.1 rather than obtained from independent human segmentation annotations. Although the authors provide a sensitivity analysis using SAM-HQ, agreement between automatic segmentation models does not necessarily establish label accuracy.
  3. "Plug-and-play" is somewhat overclaimed with respect to detector generality. SEXTANT is evaluated only on MOTIP and HAT, which share one frozen MOTIP e3 detector. The experiments therefore do not demonstrate that the query-conditioned correction transfers across detector architectures.
Questions for authors
  1. Can you provide a small human-verified mask subset on the real videos and report SEXTANT's scores against both automatic and human targets? Experiments show SAM is biased toward its own targets. To what extent does this bias affect the sim-to-real conclusions for SEXTANT ?
  2. Do the synthetic-trained heads retain their gains on real videos captured from moving platforms ? Since PanoPed-R is fixed-camera only, cross-platform transfer to real data is currently unverified.
  3. Does the detector query still carry the localization information when the detector architecture changes ? A quantitative comparison would strengthen the generality claim.
2 comments

Thank you for the positive and precise review.

Q1 / W2. Human masks and the SAM bias. We now report human-annotated masks on R-03 (General response, point 1). Against human targets, Sextant still gains 1.08 and 0.87 HOTA over U with HAT. To measure the bias, we replace the SAM-made target with an independent target and check how the IoU of each readout changes:

ReadoutSynthetic: SAM2 target to rendered maskReal R-03: SAM2.1 target to human mask
U+0.019+0.012
Sextant+0.015+0.002
MobileSAM−0.039−0.024
SAM2.1 Hiera-L−0.076−0.050

Both checks agree: the bias helps the SAM readouts, not Sextant. The sim-to-real conclusion for Sextant therefore holds. The gain also remains with SAM-HQ targets (HAT: 31.21 to 32.26 HOTA, Table 22).

Q2 / W1. Real data and moving platforms. The real test uses all 16,247 densely labeled frames, and both seeds improve every real sequence with both trackers (Sec. 5.4). We agree that real moving platforms are not verified. Within PanoPed-S, Sextant improves all four moving-camera test sequences: quadruped +0.66 to +1.13 and drone +1.86 to +2.45 HOTA (Table 17). Real sequences from a robot or drone need a new rig, new consent and an annotation effort of the same size as PanoPed-R. We list them as future work and make no claim about real cross-platform transfer.

Q3 / W3. Other detector architectures. To test the plug-in claim, we attach Sextant, with the same size and recipe, to D-FINE-L (Sec. 5.5, Table 6). Its HGNetv2 backbone and decoder differ from the Deformable DETR detector of MOTIP. PanoPed-S test, seed-42 D-FINE + OC-SORT of Table 3:

ReadoutHOTA (42 / 43)
Geometric readout27.81
Geometry head27.97 / 27.96
Sextant29.57 / 29.55

Sextant raises HOTA by 1.76 and 1.74, while the capacity-matched geometry head gains only 0.16 and 0.15. Both seeds improve all 8 test sequences. The query of a different detector architecture also carries the localization that the box loses. We also agree that "plug-and-play" was too broad. The revision calls Sextant a plug-in head for query-based trackers and states that its weights are tied to the detector it is trained on.

Thank the authors for their clarifications. My concerns have been well addressed.

Comments on the paper

1 comment
AuthorsOpenReview ↗

GENERAL RESPONSE

We thank all reviewers for their careful reading. This response answers the points raised by several reviewers. All revisions in the PDF are shown in blue.

1. Human-annotated masks on real video (ZSaF Q1, EuG8 Q1, vxGo Q2). We first clarify that Support-auto is not an unconstrained automatic label. Every mask is prompted by a human-reviewed box and generated in a local view centered on that box. These boxes come from our manual box annotation, which took us about one month. Support-auto may have small errors, but it remains statistically meaningful.

Following the reviewers' advice, we annotated masks by hand on R-03, the most representative labeled sequence: a multilevel plaza with stairs, strong changes in scale and frequent occlusion. We annotated every 30th frame, 144 frames with 1,431 pedestrian instances (new App. D.8). The SAM2.1 masks prompted by our boxes turn out to be largely accurate. The annotator kept 93% of them unchanged, and 60 masks drawn from scratch without seeing SAM2.1 agree with SAM2.1 at a median IoU of 0.90.

Tracking on these 144 frames (HAT, sim-to-real HOTA, seeds 42 / 43):

ReadoutSAM2.1 targetsHuman targets
U (geometric)41.5141.54
Geometry head42.00 / 41.9642.04 / 41.97
Sextant42.58 / 42.3742.62 / 42.41
MobileSAM44.3444.35
SAM2.1 Hiera-L48.8748.82

Against human targets, every readout keeps its rank, and Sextant still gains 1.08 and 0.87 HOTA over U (MOTIP: 1.02 and 0.77). On the masks drawn from scratch, which do not depend on SAM2.1, SAM2.1 and MobileSAM lose 0.050 and 0.024 IoU compared with SAM2.1 targets, while Sextant changes by less than 0.005. The bias of SAM-made targets therefore favors the SAM readouts, not Sextant.

2. Why real-only LOSO does not help (EuG8 Q3, vxGo Q3). In short, the LOSO drop is caused by the training data, not by the method. Real-only training sees far too few viewpoints, so the head overfits. The PanoPed-S training set has 22 fixed-camera sequences at five sites and 22 moving-camera sequences on quadrupeds and drones. The moving cameras alone give 39,600 frames, and nearly every frame has a different camera pose. The head therefore learns from tens of thousands of viewpoints. A PanoPed-R LOSO fold trains on 2 real sequences, which means only 2 fixed viewpoints and the same 10 people. Learning a correction that generalizes from two fixed views is very hard. This is why the paper does not recommend LOSO as a standalone benchmark (Sec. 3.4).

New controlled experiment (Sec. 5.5 and Table 6; App. F.4). Same frozen MOTIP detector, PanoPed-S test, no domain shift. Only the training set of the head changes.

Head training setPairsTest HOTA (42 / 43)ΔHOTA
none (geometric readout)047.300
2 fixed sequences82,73344.76 / 44.46−2.54 / −2.84
44 sequences, subsampled21,37249.04 / 48.98+1.74 / +1.68
44 sequences (paper)all49.49 / 49.47+2.19 / +2.17

Two fixed sequences make Sextant worse even in simulation. Four times fewer pairs spread over 44 sequences make it better. The number of viewpoints, not the number of samples, decides whether the head generalizes. Simulation supplies these viewpoints at low cost.

We acknowledge that, because of equipment limits, PanoPed-R does not include moving viewpoints from quadruped or drone platforms. Such real sequences are expensive to record and to label densely, and we leave them to future work.

3. Revisions (in blue in the PDF).

  • App. D.8 (new): human-annotated masks on R-03, with pointers in Sec. 3.3 and Sec. 5.4.
  • Sec. 5.5 (new): plug-in to a second detector (D-FINE-L), training data, and comparison with segmentation readouts (MobileSAM, SAM2.1). Details in App. F.4 and F.11.
  • App. F.9: why real-only LOSO training lowers HOTA.
  • Sec. 4.2: a paragraph that explains how Sextant differs from the detector's box head (vxGo).
  • Related work and Table 1: 360VOTS and BFoV/eBFoV (vxGo).
  • Abstract, introduction and conclusion: "plug-and-play" is now "plug-in head for query-based trackers" (ZSaF).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.