- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The authors introduce NuscTrack, a benchmark for tracking arbitrary 3D points across the six surround-view cameras for driving scenarios, covering both object surfaces and static background. Built from nuScenes, it generates metric 3D trajectories using LiDAR, annotated object motion, and ego poses, with per-camera visibility determined through depth completion. The dataset contains approximately 15.5 million training tracks and 3 million validation tracks, evaluated using shared pixel queries without ground-truth query depth or scale alignment. Experiments show that existing monocular and multi-view trackers transfer poorly to this setting, while training on NuscTrack substantially improves accuracy. Surround-view input particularly benefits object tracking, whereas background tracking remains challenging.
Critical strengths
-
NuscTrack evaluates metric 3D tracking across six surround-view cameras, covering both annotated objects and static background. Following the same physical point as it moves between cameras is a useful capability that monocular driving benchmarks do not directly evaluate.
-
The authors provide very useful shared evaluation protocol. All methods receive the same pixel queries without ground-truth query depth or scale alignment, and results separate object and background performance. The improvement of MVTracker after NuscTrack training demonstrates the value of domain-specific supervision, while the single-view versus surround-view comparison reveals that additional cameras benefit object tracking much more than background tracking. The paper presents a very useful TAP dataset for driving scenarios.
Critical weaknesses
-
The use of external OMNI-DC-completed LiDAR depth to decide whether another surface occludes a point will cause some errors near object boundaries, thin structures, or depth discontinuities can produce incorrect visibility labels. This may affect the evaluation results even though trackers do not receive that completed depth as input. So if we use another depth completion model, we obtain different evaluation results.
-
The benchmark uses 24-frame clips sampled at only 2 Hz, which limits its coverage of temporal tracking behavior. This has been acknowledged in the paper. Although this sampling creates challenging large displacements, the paper does not investigate how frame rate affects tracking performance or whether its findings extend to higher-frequency driving video.
Questions for authors
This is a good paper, however, with some additional clarification and analysis, it may be useful for the research community:
-
How much of the low 3D tracking accuracy comes from inaccurate depth initialization rather than tracking failures? Could the authors report an oracle-query-depth experiment to separate these sources of error by inputing the correct GT-depth to the tracking models?
-
From the paper presentation, I still not sure why does surround-view input substantially improve object tracking but barely improve background tracking? Would it be possible to analyze whether this difference comes from depth errors, texture, or point distance?
-
Could the authors break down tracking accuracy by distance from the camera? Since evaluation uses fixed metric thresholds, this would help reveal whether distant points account for much of the poor performance. Moreover, what if the object is observed in the camera but not observe in the lidar, is this the limitation of the dataset?
-
How sensitive are the visibility labels and model rankings to the chosen occlusion threshold of 0.9? If we alternate thresholds, it would substantially change the results?
-
I understand that nuScenes provides 3D box annotations at 2 Hz, which limits the availability of reliable higher-frequency tracking labels. However, I’m concerned that temporal sampling substantially affects tracking performance and the conclusions drawn from this benchmark. Could the authors explore a 10 Hz version on a small subset, using interpolated boxes from py123d library or a 3D detection-and-tracking pipeline, with manual verification of the generated labels? Even a limited comparison over the same temporal duration would help the community understand how frame rate affects model performance and whether the reported findings hold at higher sampling rates.
-
Should the name be nuScTrack?
Thank you for the clarifications and additional results.
My concern about the reliability of the labels for training and evaluation remains. The audit found location errors in 8.3% (8/96) of the sampled tracks and visibility errors in 13.5% (13/96). The audit checked 2D projections, leaving the accuracy of the 3D labels unverified. The labels also assign the same rigid motion to every point on an object, even when some parts move relative to the rest. A tracker that correctly follows an articulated part can therefore receive a lower score for disagreeing with the label. This limits the benchmark’s value for measuring progress in 3D point tracking.
I retain my original rating of 2 (weak rejection).