NuscTrack: A Surround-View Benchmark for 3D Track-Any-Point in Autonomous Driving

Track-Any-Point (TAP) follows a queried pixel through a video with occlusion labels. On a surround-view driving rig, the same physical point often leaves the query camera's field of view and must be recovered in another camera, even when it lies on unlabeled background that has no 3D box. Existing autonomous driving Tracking benchmarks do not measure that setting: they use one video stream or indoor cameras with large overlap, and driving TAP has been mostly monocular and instance-centric. We introduce NuscTrack, a surround-view benchmark for cross-view 3D TAP on nuScenes: a queried pixel in one camera maps to a metric ego-frame trajectory and per-camera visibility. Instance-surface points follow the rigid motion of their 3D boxes, static background points remain fixed in the world frame, and visibility is determined using completed LiDAR depth. The processed pool has $15.5$M / $3.0$M unique 3D tracks (train / val) after length, visibility, and range filters; $78\%$ are visible in at least two cameras, covering both instance surfaces and unlabeled background. We evaluate all methods on the same image queries, reporting 3D-AJ, APD, MTE, and OA split by All / instance / background, without fitting scale to ground truth. Models trained on monocular TAP data or on indoor multi-view data do not transfer to this setting: they remain at single-digit 3D-AJ with multi-metre error. Training the same multi-view model on NuscTrack improves metric localization, while background queries stay harder than instance queries. On the same queries, six-camera input improves instance 3D-AJ over a single camera while background 3D-AJ barely changes. We hope NuscTrack can serve as a shared benchmark for metric 3D TAP on surround-view driving video, measuring whether a tracker keeps an identity across cameras on both instance surfaces and unlabeled background.