acceptodds
Under review as a conference paper at ICLR 2027

NuscTrack: A Surround-View Benchmark for 3D Track-Any-Point in Autonomous Driving

Abstract

Track-Any-Point (TAP) follows a queried pixel through a video with occlusion labels. On a surround-view driving rig, the same physical point often leaves the query camera's field of view and must be recovered in another camera, even when it lies on unlabeled background that has no 3D box. Existing autonomous driving Tracking benchmarks do not measure that setting: they use one video stream or indoor cameras with large overlap, and driving TAP has been mostly monocular and instance-centric. We introduce NuscTrack, a surround-view benchmark for cross-view 3D TAP on nuScenes: a queried pixel in one camera maps to a metric ego-frame trajectory and per-camera visibility. Instance-surface points follow the rigid motion of their 3D boxes, static background points remain fixed in the world frame, and visibility is determined using completed LiDAR depth. The processed pool has M / M unique 3D tracks (train / val) after length, visibility, and range filters; are visible in at least two cameras, covering both instance surfaces and unlabeled background. We evaluate all methods on the same image queries, reporting 3D-AJ, APD, MTE, and OA split by All / instance / background, without fitting scale to ground truth. Models trained on monocular TAP data or on indoor multi-view data do not transfer to this setting: they remain at single-digit 3D-AJ with multi-metre error. Training the same multi-view model on NuscTrack improves metric localization, while background queries stay harder than instance queries. On the same queries, six-camera input improves instance 3D-AJ over a single camera while background 3D-AJ barely changes. We hope NuscTrack can serve as a shared benchmark for metric 3D TAP on surround-view driving video, measuring whether a tracker keeps an identity across cameras on both instance surfaces and unlabeled background.

Primary area:
datasets and benchmarks
Keywords:
track any point, 3D point tracking, multi-view tracking, surround-view, autonomous driving, datasets and benchmarks, nuScenes, correspondence

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_28vDOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors introduce NuscTrack, a benchmark for tracking arbitrary 3D points across the six surround-view cameras for driving scenarios, covering both object surfaces and static background. Built from nuScenes, it generates metric 3D trajectories using LiDAR, annotated object motion, and ego poses, with per-camera visibility determined through depth completion. The dataset contains approximately 15.5 million training tracks and 3 million validation tracks, evaluated using shared pixel queries without ground-truth query depth or scale alignment. Experiments show that existing monocular and multi-view trackers transfer poorly to this setting, while training on NuscTrack substantially improves accuracy. Surround-view input particularly benefits object tracking, whereas background tracking remains challenging.

Critical strengths
  • NuscTrack evaluates metric 3D tracking across six surround-view cameras, covering both annotated objects and static background. Following the same physical point as it moves between cameras is a useful capability that monocular driving benchmarks do not directly evaluate.

  • The authors provide very useful shared evaluation protocol. All methods receive the same pixel queries without ground-truth query depth or scale alignment, and results separate object and background performance. The improvement of MVTracker after NuscTrack training demonstrates the value of domain-specific supervision, while the single-view versus surround-view comparison reveals that additional cameras benefit object tracking much more than background tracking. The paper presents a very useful TAP dataset for driving scenarios.

Critical weaknesses
  • The use of external OMNI-DC-completed LiDAR depth to decide whether another surface occludes a point will cause some errors near object boundaries, thin structures, or depth discontinuities can produce incorrect visibility labels. This may affect the evaluation results even though trackers do not receive that completed depth as input. So if we use another depth completion model, we obtain different evaluation results.

  • The benchmark uses 24-frame clips sampled at only 2 Hz, which limits its coverage of temporal tracking behavior. This has been acknowledged in the paper. Although this sampling creates challenging large displacements, the paper does not investigate how frame rate affects tracking performance or whether its findings extend to higher-frequency driving video.

Questions for authors

This is a good paper, however, with some additional clarification and analysis, it may be useful for the research community:

  • How much of the low 3D tracking accuracy comes from inaccurate depth initialization rather than tracking failures? Could the authors report an oracle-query-depth experiment to separate these sources of error by inputing the correct GT-depth to the tracking models?

  • From the paper presentation, I still not sure why does surround-view input substantially improve object tracking but barely improve background tracking? Would it be possible to analyze whether this difference comes from depth errors, texture, or point distance?

  • Could the authors break down tracking accuracy by distance from the camera? Since evaluation uses fixed metric thresholds, this would help reveal whether distant points account for much of the poor performance. Moreover, what if the object is observed in the camera but not observe in the lidar, is this the limitation of the dataset?

  • How sensitive are the visibility labels and model rankings to the chosen occlusion threshold of 0.9? If we alternate thresholds, it would substantially change the results?

  • I understand that nuScenes provides 3D box annotations at 2 Hz, which limits the availability of reliable higher-frequency tracking labels. However, I’m concerned that temporal sampling substantially affects tracking performance and the conclusions drawn from this benchmark. Could the authors explore a 10 Hz version on a small subset, using interpolated boxes from py123d library or a 3D detection-and-tracking pipeline, with manual verification of the generated labels? Even a limited comparison over the same temporal duration would help the community understand how frame rate affects model performance and whether the reported findings hold at higher sampling rates.

  • Should the name be nuScTrack?

Reviewer_kt3KOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes NuscTrack, a benchmark for 3D Track-Any-Point on the six-camera nuScenes rig: a pixel query in one camera is to be tracked as a metric ego-frame 3D trajectory with per-camera visibility, including on unlabeled static background. Labels are generated automatically by propagating LiDAR points with 3D box poses (instances) or ego poses (background), with visibility derived from completed LiDAR depth; a small human spot check is included. The stated contributions are the task definition, the dataset, and a shared-query evaluation protocol with fixed-metre thresholds and camera-level visibility metrics. As validation, the paper benchmarks monocular trackers, an indoor multi-view tracker and two BEV adaptations, and reports that zero-shot models transfer poorly, that training on NuscTrack helps, and that surround-view input helps instance points more than background. No new tracking method is proposed.

Critical strengths
  1. The task itself is well chosen. According to the paper's survey (Sec. 1-2, Tables 1-2), no existing TAP resource scores whether a queried point keeps its identity after leaving the query camera on a surround driving rig, for both boxed objects and background without a 3D box. The gap is genuine, and framing it as a metric ego-frame task on a public six-camera dataset is the right setting for driving. This is the main reason I do not rate the paper lower.

  2. The protocol is disciplined and honestly reported. One shared 2D query list, no ground-truth 3D or query depth, one-pass All / instance / background splits, fixed-metre thresholds without scale fitting, camera-IoU separated from any-view OA, and disclosed unmatched pretraining (Sec. 3; App. A; Sec. 5). Because the protocol is sound, the label-quality concern below may be addressable without changing the query protocol.

  3. Two useful controls exist. Same-initialization zero-shot vs. finetuned MVTracker (Table 4; App. B), matched CAM_FRONT queries for single- vs. surround-view (Table 6), plus a label spot check, limitations and a dataset card (App. C, F, G). The in-domain-training result in Table 4 therefore rests on a clean comparison.

Critical weaknesses
  1. The dataset applies DriveTrack's labelling recipe to nuScenes with incremental additions, and the new findings it reports are limited.

    • Recipe. By the authors' own account, densify-and-warp is DriveTrack's (Sec. 4.1, Fig. 3); what is new is projecting to six cameras with per-camera visibility, adding lidarseg background, and replacing nearest-neighbour depth with OMNI-DC, all on nuScenes only (App. C). None of this is wrong, but it means that most of the paper's value must come from the task, the protocol and what the benchmark reveals.
    • Finding 1, "zero-shot models do not transfer". TAPVid-3D already showed that monocular 3D-AJ collapses on driving video, and the paper cites this (p. 1). More importantly, the evaluation changes three things at once relative to the models' training data and usual evaluation - 2 Hz keyframes (App. C), per-camera UniDepthV2 depth (Sec. 5), and fixed-metre thresholds without alignment (Sec. 3) - so the paper cannot yet say how much of the drop is due to the surround-view setting itself. Part of it is expected by construction, because the monocular rows see only the query camera (Sec. 5).
    • Finding 2, "background is harder". This holds for 3D-AJ (one row ties) but not for MTE, which is lower on background in three rows of Table 4. A plausible reason, which the paper mentions only briefly (App. D): background points are fixed in the world (Sec. 4.1), so their ego-frame trajectory is determined by the query depth and ego motion alone; the background split may therefore be measuring depth lifting as much as tracking (background is also harder in 2D: Table 12). Without that analysis, the cause of the gap is unclear.
    • Finding 3, cross-camera handover. This is the stated motivation (Sec. 1) and the reason a surround benchmark is needed, yet no aggregate number isolates it: Table 4 averages over all frames, and the App. I scores are whole-clip scores for selected clips. In aggregate, the reader cannot tell whether trained models keep identity across a camera boundary or simply localize better before reaching it.
    • Internal contradiction. p. 8 ll. 404-406 says the BEV adaptations "sit with that trained setting rather than with the zero-shot rows", but in Table 4 two of the three BEV rows (2.78 and 2.53 3D-AJ) are below every zero-shot row (5.07-7.83). The error is small, but it is in the main results text.
    • Why consequential: with limited novelty in the data, the contribution has to come from well-supported findings. As written, the surround-specific findings are partly known, confounded, metric-dependent or not isolated; only the in-domain-training gain rests on a clean comparison.
  2. Label precision is not shown to support the metric.

    • The metric averages five thresholds (App. A); two of them (0.01 and 0.04 m) are at centimetre scale, and a third (0.16 m) is about the audit's own tolerance.
    • The only validation is a 96-track, three-timestamp spot check with a 12-pixel tolerance (App. F). At the mean range the paper puts that at about the 0.16 m bin, or coarser if the audit used the 432x768 model resolution, which the paper does not state. The check therefore says nothing about the two finest bins.
    • It is a 2D check on stored projections, so it is largely insensitive to error along the viewing ray. Uncertain counts and per-split breakdowns (instance/background, rigid/non-rigid) are not reported, so it is unclear where the 8 location and 13 visibility failures occur.
    • Other error sources are acknowledged but not quantified: the rigid-box approximation for humans and the dependence of visibility on completed depth and calibration (App. C), and LiDAR-camera time offsets, which Sec. 4.1 mitigates for ego poses only.
    • Why consequential: a score at a given threshold is interpretable only if label error is below that threshold. Until this is shown, the finest bins may be measuring label error, and it is not shown that differences between methods at 10-13 3D-AJ are robust to it.
Questions for authors

Questions whose answers could change my assessment

  1. (W2) Please report 3D-AJ and APD per threshold for the main rows of Table 4, and an estimate of label error in metres on the audited tracks where a geometric check is possible. Because every row's All MTE is at least 3.81 m (Table 4), near-zero scores at 0.01 and 0.04 m may occur even with perfect labels, so they would not by themselves show label noise. A label-error estimate below those thresholds would resolve the localisation part of W2.
  2. (W2) For the App. F audit: how many tracks were marked Uncertain; how do the 8 location and 13 visibility failures split by instance/background and by vehicle/human/movable; what image resolution was used for the 12-pixel tolerance; and how many auditors were there?
  3. (W1, handover) Using the existing validation predictions and visibility labels, please report 3D-AJ and MTE (and camera-IoU where a six-camera visibility head exists) separately on (a) frames where the query camera no longer sees the point but another camera does and (b) frames where the query camera still sees it, with frame counts, for the trained rows of Table 4 and for the two matched rows of Table 6. This is the measurement the paper is motivated by (Sec. 1, App. I) and it requires no new training.
  4. (W1, zero-shot) If the oracle query-depth setting in App. B (qp_manner=use_gt) was evaluated, please report it on the same queries. If possible, also report scores with TAPVid-3D's depth-scaled thresholds. These would indicate how much of the zero-shot drop comes from query depth and from the metric choice.
  5. (W1, Table 6) What was the single-view model trained on (CAM_FRONT only, or all cameras with single-view input?), and with which initialization? How are frames visible only in other cameras scored for it? How was the CAM_FRONT query list (n=112,807) sampled, and does it use the same seed and stratification as the 115,200-query list of Table 4? Are ego poses provided at test time; if so, is a background trajectory not fully determined by the query depth, so that the background split measures depth lifting rather than tracking?
  6. (W2, visibility) In the visibility rule, are "completed depth" and "point-to-camera distance" the same quantity (both axial depth or both range)? Sec. 4.1 reads as a range; OMNI-DC outputs axial depth. If the two quantities differ, this would affect the visibility labels and W2.

Suggestions for future work (not score-relevant)

  • Results stratified by range, to separate the instance/background gap from distance.
  • Scene-level bootstrap confidence intervals for Tables 4 and 6, and a statement of whether the two clips per scene overlap.
  • A second surround-view source (Waymo or Argoverse 2) to show the recipe is not nuScenes-specific; absence of this does not affect my rating.
Reviewer_rfssOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces NuscTrack, a dataset and evaluation protocol for surround-view 3D point tracking built on nuScenes. Given surround-view video, camera calibration, and a 2D pixel query, the task is to predict the corresponding physical point’s subsequent 3D trajectory and visibility in each camera. Instance tracks are generated by propagating LiDAR points using the annotated object boxes’ 6-DoF poses (3D translation and rotation), while background points remain fixed in world coordinates. OMNI-DC completes LiDAR-projected depth, and a fixed 0.9 ratio threshold determines occlusion labels. The experiments evaluate existing trackers and two BEV adaptations, and compare single-camera with six-camera input.

Critical strengths
  • The surround-view setting is relevant. A point may leave the query camera while remaining visible in another camera. Table 6 compares single-camera and six-camera input on the same queries, improving instance 3D-AJ from 5.8 to 12.1. This provides direct evidence for the value of surround-view input. Including background points without object boxes also broadens coverage beyond instance surfaces.
  • The shared queries and per-camera evaluation are useful. Methods receive the same 2D pixel queries, without ground-truth query depth or scale alignment to ground-truth trajectories. Results are reported separately for instances and background. Camera-IoU checks which cameras see the physical point specified by a query pixel, providing more information than whether that point is visible in at least one camera.
Critical weaknesses
  • Annotation quality remains a major concern. Appendix F inspects only 96 tracks, mainly at three timestamps per track, yet finds 8 tracks with location errors and 13 with visibility errors; these groups may overlap. The paper does not explain their causes or whether they have been corrected. The audit checks image-plane alignment but does not directly measure depth error. A correct image location does not guarantee correct depth. Therefore, this audit does not establish centimetre-level accuracy of the 3D labels, although evaluation includes 1 cm and 4 cm error thresholds.

  • The reliability of depth and boundary labels is insufficiently supported. OMNI-DC directly affects visibility labels, but the paper does not report its depth or occlusion errors on nuScenes, or compare it with newer depth methods such as DA3. The choice of the 0.9 threshold needs justification, particularly at object boundaries. Grazing occlusions, glass, and thin poles are categorized as Uncertain and excluded from failure counts, but their numbers and treatment during training and evaluation are not reported. D4RT deliberately samples 30% of its spatial training queries near depth or motion boundaries (Appendix A). Such samples may provide important supervision, making the reliability of these labels particularly relevant.

  • Motion coverage is limited, and the camera configuration needs further context. Door opening, dump-bed lifting, and excavator-arm motion cannot be represented by a single rigid transformation of the whole object box. A tracker that follows these parts correctly could therefore be penalized. These labels do not evaluate tracking of articulated mechanical motion. Furthermore, nuScenes cameras are rigidly mounted, with fixed relative positions and orientations within each clip. The paper should explain how representative this camera count, view coverage, and relative orientation are of other autonomous-driving platforms, and clarify the scope of its conclusions.

  • Minor issues: “valuation set” in Table 6 should be “validation set.” Figure 5 has crowded labels and caption text, and an extra comma in “0.5–0.9,M.”

Questions for authors
  • What caused the 8 location failures and 13 visibility errors in the existing audit? Have they been corrected? How many cases were marked Uncertain, and how are their labels handled during training and evaluation?
  • How was the 0.9 occlusion threshold chosen? What evidence supports its reliability at object boundaries and thin structures? Were newer depth methods such as DA3 compared?
  • How representative is nuScenes’ six-camera setup of camera configurations used in autonomous driving? To what extent do the findings apply to other vehicle platforms?
  • Given that the labels assume rigid object motion and a static background, could a simple geometric method, using the same test-time inputs, outperform the learned trackers at lower computational cost and without NuscTrack-specific training? What evidence shows that this benchmark requires the capabilities of general-purpose point trackers?
1 comment

Thank you for the clarifications and additional results.

My concern about the reliability of the labels for training and evaluation remains. The audit found location errors in 8.3% (8/96) of the sampled tracks and visibility errors in 13.5% (13/96). The audit checked 2D projections, leaving the accuracy of the 3D labels unverified. The labels also assign the same rigid motion to every point on an object, even when some parts move relative to the rest. A tracker that correctly follows an articulated part can therefore receive a lower score for disagreeing with the label. This limits the benchmark’s value for measuring progress in 3D point tracking.

I retain my original rating of 2 (weak rejection).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.