- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
TrackEverything studies long-horizon, all-frame dense 3D point tracking from RGB videos and externally supplied geometry. It combines cross-window voxel merging, dynamic-only trajectory decoding, and 3D WAFT feature sampling. Experiments on TAPVid-3D, Dynamic Replica, and PointOdyssey report competitive accuracy and scalability, including sequences exceeding 1,000 frames within 40 GB of GPU memory. Its main contribution is a persistent representation that reduces redundant tracking state across frames.
Critical strengths
-
The paper tackles a meaningful point tracking scaling problem: retaining tracks for surfaces observed over a long video without maintaining a separate representation for each observation. Cross-window merging addresses the growth of persistent state, while endpoint-first prediction limits full trajectory decoding to points classified as dynamic. The ablations support the efficiency benefit of this decomposition.
-
The evaluation covers multiple geometry sources, short and long sequences, and comparisons with sparse and dense trackers. It provides evidence that all-frame coverage can be maintained at a computational cost that was impractical for the evaluated all-frame dense baselines.
Critical weaknesses
-
The incremental benefit of the separate visual encoder and 3D WAFT is not sufficiently isolated from existing tracking designs. The paper also leaves open the question of how errors in the input geometry, and self-occlusion affect persistent point correspondence. These issues motivate Questions 1(a)–1(e).
-
The experiments support the method’s scalability, but leave important limits of the accuracy and efficiency claims unclear. The large change in accuracy when ground-truth geometry is used, the workloads included in the runtime comparisons, and the different points displayed in the qualitative comparisons warrant closer examination. These issues motivate Questions 2(a)–2(c).
Questions for authors
-
Several aspects of the representation and its design need clarification:
(a) TrackEverything uses pointmaps from a geometry predictor but runs a separate DINOv3 encoder for tracking features. Why are features from the geometry backbone insufficient?
(b) TrackEverything combines a world-coordinate feature cloud with feature sampling at predicted positions. TAPIP3D (NeurIPS 2025) also uses a world-coordinate feature cloud, while WAFT (ICLR 2026) and CoWTracker (CVPR 2026) use feature warping. What does 3D WAFT add beyond these existing ideas? The “without WAFT” ablation measures its effect within TrackEverything. A comparison using the same input geometry and query points would make the benefit of its matching design clearer.
(c) When an object rotates, newly visible surfaces can contribute new points. What happens to points that were tracked before becoming hidden? If they reappear, how does the method continue their original tracks instead of creating new ones or merging them with nearby points?
(d) TrackEverything uses estimated geometry to place points in 3D and visual features to refine their motion. (1) If a pointmap or camera pose is wrong, can visual evidence correct the correspondence before a false voxel merge occurs? (2) Could establishing correspondence primarily from features, then using geometry for 3D placement and merging, reduce these errors? VGGT (CVPR 2025) has shown that its pretrained features can support point tracking. GOT-Edit (ICLR 2026) also provides an example of explicitly preserving semantic discrimination when incorporating geometric cues. Results grouped by geometry error would clarify how the two signals interact in TrackEverything.
(e) Cross-window merging pools observations of a surface that may have been seen from different viewpoints. What appearance information does the merged track retain, and which feature serves as the source feature for 3D WAFT in later windows? Does matching become less accurate after a large viewpoint change or after several merges?
-
Several experimental details would clarify the scope of the results:
(a) In Table 3, replacing estimated Pi3 geometry with GT geometry raises TrackEverything’s APD-P from 36.6 to 85.2 on Dynamic Replica and from 24.9 to 73.2 on PointOdyssey. DeltaV2 also improves, but by a smaller margin. What explains the larger change for TrackEverything?
(b) Figure 3 compares runtime and memory on an L40S GPU, but the methods may track different numbers of points. For each method, does the reported runtime include geometry prediction, and how many points are tracked over the video, including points added after the first frame?
(c) In the supplementary figures and the project website, TrackEverything and the baselines visualize different sets of points. The captions state that only dynamic trajectories are shown. How were the points and trajectories selected for each method? Could the authors show all methods tracking the same points from the first frame, so that correspondence quality can be compared directly.
9 comments
Harmful merges (% of merged pairs, lower is better)
| Setting | Correspondence | Geometry | POD, 32 f | DR, 32 f | POD, 128 f | DR, 128 f |
|---|---|---|---|---|---|---|
| (a) | Ours | GT | 0.01% | 0.00% | 0.20% | 0.06% |
| (b) | Ours | Estimated (VGGT-Ω) | 0.11% | 0.06% | 0.72% | 0.18% |
| (c) | Perfect 2D (GT tracks) | Estimated (VGGT-Ω) | 0.12% | 0.03% | 0.36% | 0.06% |
| (d) | CoTracker3 2D tracks | Estimated (VGGT-Ω) | 0.20% | 0.03% | – | – |
32 f: 32 frames. 128 f: 128 frames
Here are our conclusions:
(i) Harmful merges are rare even in the full setting with estimated geometry: 0.11% / 0.06% (POD / DR) on 32-frame videos, and 0.72% / 0.18% on 128-frame videos.
(ii) Most of them are caused by geometry errors. With GT geometry, harmful merges drop to 0.01% / 0.00% (32 frames) and 0.20% / 0.06% (128 frames). Errors in the input geometry therefore account for roughly 70% or more of the harmful merges. Even with perfect correspondence, estimated geometry still yields more harmful merges than our model with GT geometry on POD (0.36% vs. 0.20%).
(iii) Perfect correspondence recovers part of the error, but learned 2D correspondence gives no consistent gain. Perfect 2D tracks halve harmful merges on 128-frame videos (0.72% → 0.36% POD, 0.18% → 0.06% DR). CoTracker3's learned 2D correspondences nearly double harmful merges on POD (0.20% vs. 0.11%) and halve them on DR (0.03% vs. 0.06%), while keeping about 1.5× more tokens on both (thus failing to optimally de-duplicate).
We added this analysis in Sec. A.8 and Table 5b of the supplementary (in blue).
What appearance information does the merged track retain, and which feature serves as the source feature for 3D WAFT in later windows? Does matching become less accurate after a large viewpoint change or after several merges?
We mean-pool the appearance features of the merged points, and this pooled feature serves as the 3D WAFT source feature in later windows. Like all trackers, ours gradually loses accuracy with video length and viewpoint change, but merging adds little further degradation. On full-length videos in Table 1, TrackEverything remains competitive with state-of-the-art sparse trackers that perform no merging, and shows similar degradation in performance when going from 48 frames to 128 frames. We also measured merge quality directly on 128-frame videos: the fraction of harmful merges grows only slowly with the number of window boundaries, from 0.21% to 0.72% of point pairs on PointOdyssey and from 0.08% to 0.18% on Dynamic Replica over seven boundaries (Supp. Sec. A.8, Table 5a).
In Table 3, replacing estimated Pi3 geometry with ground-truth geometry raises TrackEverything’s APD-P from 36.6 to 85.2 on Dynamic Replica and from 24.9 to 73.2 on PointOdyssey. DeltaV2 also improves, but by a smaller margin. What explains the larger change for TrackEverything?
While it is hard to pinpoint the exact mechanism, we believe the difference lies in coordinate system differences. TrackEverything attends over a world-coordinate 3D feature cloud, so it can directly benefit from more accurate point clouds and camera poses. DeltaV2 operates in camera coordinates, so it cannot fully exploit accurate camera poses even when they are available.
Figure 3 compares runtime and memory on an L40S GPU, but the methods may track different numbers of points. For each method, does the reported runtime include geometry prediction, and how many points are tracked over the video, including points added after the first frame? Reporting these numbers would make the speed comparison easier to interpret.
Thank you for pointing out these ambiguities.
Geometry. In the submitted version, Fig. 3 reported tracking cost only. We have updated Fig. 3 and Fig. 11 to also include the inference time and memory of geometry estimation. All trends remain similar, and the original tracking-only figure is now Fig. 12.
Number of points. Fig. 3d–f controls for this: the x-axis is the number of query points (100–10k), matched across methods. The baselines track exactly these queries. TrackEverything tracks the same queries plus every point in every frame, including points that appear after the first frame, since its dense tracking cannot be switched off. Even so, its latency and memory stay low and grow slowly with the number of queries. For reference, a 400-frame PointOdyssey video holds ~70k persistent tracks after de-duplication, versus 4.9M if every pixel of every frame were tracked.
In the supplementary figures and the project website, TrackEverything and the baselines visualize different sets of points. The captions also state that only dynamic trajectories are shown. How were the points and trajectories selected for each method? Could the authors show all methods tracking the same points from the first frame, with identical visibility and rendering settings, so that correspondence quality can be compared directly?
Point selection. Our original visualizations were designed to highlight a new capability: TrackEverything efficiently tracks all points across all frames of a video, rather than only first-frame points or a sparse set of queries. It therefore needs no point sampling. For the baselines, we sampled query points on a grid in the first frame. Denser grids, as well as grids sampled on every frame, ran out of memory.
Same points, same rendering. Following your suggestion, we added a controlled comparison (Rebuttal section of https://iclr-8302.netlify.app/ and Fig. 8 in the appendix). Every method tracks the same 40×40 grid sampled on the first frame, and all tracks are drawn with the same renderer. DeltaV2, CoTracker3, SpatialTracker-v2 and TrackEverything all perform well, with slightly cleaner tracks for TrackEverything, while VDPM struggles noticeably. This matches Table 1, where TrackEverything is competitive with sparse and first-frame dense trackers and significantly outperforms VDPM. The key advantage of TrackEverything is that it delivers this accuracy while tracking every point across long videos.
Thank you again for a thoughtful and detailed review that has made the paper stronger. We believe the revision addresses your main concerns, in particular the new analysis separating geometry and correspondence errors in de-duplication (Sec. A.8), the matched same-points comparison (Fig. 8 and website), the tempered 3D WAFT claims, and the experiment with VGGT-Ω features. If so, we would be grateful if you would consider raising your score, or let us know what concerns remain.
They are not insufficient. Motivated by your question, we retrained our model with the DINOv3 ViT replaced by the VGGT-Ω backbone, keeping everything else identical. Due to time constraints, we train both variants for a shorter schedule (128k iterations, without iterative refinement). The two perform on par: 25.7 vs. 26.5 average 3D APD on our TAPVid-3D validation splits (VGGT-Ω vs. DINOv3), and 60.7 vs. 62.4 on Dynamic Replica. TrackEverything can therefore use the geometry backbone's features when trained with them. We keep a dedicated visual encoder so that the tracker works with any geometry source, including ground truth, rather than being tied to one geometry model's features. This encoder can itself be any pretrained backbone, including the geometry model's.
TAPIP3D also uses a world-coordinate feature cloud, but matches each query through 3D kNN feature correlations in every frame, whose cost grows with neighbours × queries × frames, so replacing it with cheap warp based feature comparisons provide some significant savings. Meanwhile, we differentiate from WAFT and CoWTracker primarily by adapting this operation to 3D. That said, we have now tempered our WAFT-related novelty claims. The contribution statement now says we "adapt" warping-based feature sampling to 3D rather than "introduce" 3D WAFT, and the abstract, introduction and conclusion no longer list it among our key innovations. It nevertheless remains a critical design choice: it is the matching operator that lets us refine all points at once and scale to all-frame tracking.
With regard to making a comparison using the same input geometry and query points, Table 1 actually already provides this. TAPIP3D is given the same VGGT-Ω pointmaps and the same TAPVid-3D query points as TrackEverything. TrackEverything is on par or better (APD-P 34.7 vs. 34.1 on the first 48 frames, 31.2 vs. 29.8 on full-length videos) while also tracking every other point in the video. Within our model, removing 3D WAFT lowers APD-P from 31.2 to 29.3 (Table 2). CoWTracker is a 2D tracker without geometry input, so TAPIP3D is the matched-input comparison.
Great question. The key is that hidden points are not dropped, they persist through the occlusion and receive position & visibility updates just like visible points. The model is trained to track through occlusion. Then, when the surface reappears, it gives birth to a new copy, and de-duplication merges the two copies if they agree at their predicted end-of-window positions.
Thank you for these questions. To separate the roles of geometry and correspondence in wrong merges, we measure 'harmful' merges i.e. merged pairs whose ground-truth motion later diverges by more than 0.04 units. We study four settings: (a) Our model with GT geometry; the harmful merges here are due only to wrong correspondence (b) Our model with estimated geometry (c) Perfect 2D correspondence (GT 2D tracks) lifted with the estimated geometry; here merge errors are due only to geometry. (d) Same as c but but with learned 2D correspondences using CoTracker3’s 2D tracks. We run these tests on PointOdyssey (POD) and Dynamic Replica (DR) where we have near-dense GT tracks. We run it on 128 frame long videos (7 window merges) and also on 32 frame windows. Tracking every pixel of every frame with CoTracker3 is expensive (~400k query points for a 32-frame video, 20 mins/video), so we run (d) on 32-frame videos only.