acceptodds
Under review as a conference paper at ICLR 2027

TrackEverything: Long Horizon Dense Tracking via De-Duplicating 3D Scene Representations

11 views

Abstract

Existing point tracking models face a fundamental tradeoff: they can either track a sparse set of query points over long horizons, or track all points across only short clips. We introduce TrackEverything, a 3D point tracker that breaks this trade-off by representing videos as persistent 3D scene tracks in world coordinates. Grounded in the insight that videos are 2D projections of an underlying 3D world, TrackEverything decouples model complexity from video duration, allowing it to scale with unique physical scene geometry instead. Our approach introduces three key innovations. First, we employ a voxelization-based de-duplication mechanism at sliding-window boundaries to merge co-located tracks, preventing repeated observations of the same surface from redundantly accumulating. Second, we decompose tracking into an endpoint refiner that predicts each point's destination and static-versus-dynamic classification, followed by a lightweight trajectory refiner that decodes dense trajectories exclusively for dynamic points. Third, we propose 3D WAFT, replacing memory-prohibitive 4D correlation volumes with efficient feature sampling in the scene cloud. To the best of our knowledge, TrackEverything is the first 3D tracker capable of tracking all visible points across videos exceeding 1000 frames within 40 GB of GPU memory. On TAPVid-3D, TrackEverything outperforms all open-source all-frame dense 3D trackers by more than 20% APD on short clips, while remaining competitive with state-of-the-art sparse trackers on long sequences, despite tracking far more points.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Point Tracking, 3D Representations

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_3SSEOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

TrackEverything studies long-horizon, all-frame dense 3D point tracking from RGB videos and externally supplied geometry. It combines cross-window voxel merging, dynamic-only trajectory decoding, and 3D WAFT feature sampling. Experiments on TAPVid-3D, Dynamic Replica, and PointOdyssey report competitive accuracy and scalability, including sequences exceeding 1,000 frames within 40 GB of GPU memory. Its main contribution is a persistent representation that reduces redundant tracking state across frames.

Critical strengths
  1. The paper tackles a meaningful point tracking scaling problem: retaining tracks for surfaces observed over a long video without maintaining a separate representation for each observation. Cross-window merging addresses the growth of persistent state, while endpoint-first prediction limits full trajectory decoding to points classified as dynamic. The ablations support the efficiency benefit of this decomposition.

  2. The evaluation covers multiple geometry sources, short and long sequences, and comparisons with sparse and dense trackers. It provides evidence that all-frame coverage can be maintained at a computational cost that was impractical for the evaluated all-frame dense baselines.

Critical weaknesses
  1. The incremental benefit of the separate visual encoder and 3D WAFT is not sufficiently isolated from existing tracking designs. The paper also leaves open the question of how errors in the input geometry, and self-occlusion affect persistent point correspondence. These issues motivate Questions 1(a)–1(e).

  2. The experiments support the method’s scalability, but leave important limits of the accuracy and efficiency claims unclear. The large change in accuracy when ground-truth geometry is used, the workloads included in the runtime comparisons, and the different points displayed in the qualitative comparisons warrant closer examination. These issues motivate Questions 2(a)–2(c).

Questions for authors
  1. Several aspects of the representation and its design need clarification:

    (a) TrackEverything uses pointmaps from a geometry predictor but runs a separate DINOv3 encoder for tracking features. Why are features from the geometry backbone insufficient?

    (b) TrackEverything combines a world-coordinate feature cloud with feature sampling at predicted positions. TAPIP3D (NeurIPS 2025) also uses a world-coordinate feature cloud, while WAFT (ICLR 2026) and CoWTracker (CVPR 2026) use feature warping. What does 3D WAFT add beyond these existing ideas? The “without WAFT” ablation measures its effect within TrackEverything. A comparison using the same input geometry and query points would make the benefit of its matching design clearer.

    (c) When an object rotates, newly visible surfaces can contribute new points. What happens to points that were tracked before becoming hidden? If they reappear, how does the method continue their original tracks instead of creating new ones or merging them with nearby points?

    (d) TrackEverything uses estimated geometry to place points in 3D and visual features to refine their motion. (1) If a pointmap or camera pose is wrong, can visual evidence correct the correspondence before a false voxel merge occurs? (2) Could establishing correspondence primarily from features, then using geometry for 3D placement and merging, reduce these errors? VGGT (CVPR 2025) has shown that its pretrained features can support point tracking. GOT-Edit (ICLR 2026) also provides an example of explicitly preserving semantic discrimination when incorporating geometric cues. Results grouped by geometry error would clarify how the two signals interact in TrackEverything.

    (e) Cross-window merging pools observations of a surface that may have been seen from different viewpoints. What appearance information does the merged track retain, and which feature serves as the source feature for 3D WAFT in later windows? Does matching become less accurate after a large viewpoint change or after several merges?

  2. Several experimental details would clarify the scope of the results:

    (a) In Table 3, replacing estimated Pi3 geometry with GT geometry raises TrackEverything’s APD-P from 36.6 to 85.2 on Dynamic Replica and from 24.9 to 73.2 on PointOdyssey. DeltaV2 also improves, but by a smaller margin. What explains the larger change for TrackEverything?

    (b) Figure 3 compares runtime and memory on an L40S GPU, but the methods may track different numbers of points. For each method, does the reported runtime include geometry prediction, and how many points are tracked over the video, including points added after the first frame?

    (c) In the supplementary figures and the project website, TrackEverything and the baselines visualize different sets of points. The captions state that only dynamic trajectories are shown. How were the points and trajectories selected for each method? Could the authors show all methods tracking the same points from the first frame, so that correspondence quality can be compared directly.

9 comments

TrackEverything uses pointmaps from a geometry predictor but runs a separate DINOv3 encoder for tracking features. Why are features from the geometry backbone insufficient?

They are not insufficient. Motivated by your question, we retrained our model with the DINOv3 ViT replaced by the VGGT-Ω backbone, keeping everything else identical. Due to time constraints, we train both variants for a shorter schedule (128k iterations, without iterative refinement). The two perform on par: 25.7 vs. 26.5 average 3D APD on our TAPVid-3D validation splits (VGGT-Ω vs. DINOv3), and 60.7 vs. 62.4 on Dynamic Replica. TrackEverything can therefore use the geometry backbone's features when trained with them. We keep a dedicated visual encoder so that the tracker works with any geometry source, including ground truth, rather than being tied to one geometry model's features. This encoder can itself be any pretrained backbone, including the geometry model's.

TAPIP3D uses world coordinate feature cloud, while WAFT and CoWTracker uses feature warping. What does 3D WAFT add beyond these existing ideas? The “without WAFT” ablation measures its effect within TrackEverything. A comparison using the same input geometry and query points would make the benefit of its matching design clearer.

TAPIP3D also uses a world-coordinate feature cloud, but matches each query through 3D kNN feature correlations in every frame, whose cost grows with neighbours × queries × frames, so replacing it with cheap warp based feature comparisons provide some significant savings. Meanwhile, we differentiate from WAFT and CoWTracker primarily by adapting this operation to 3D. That said, we have now tempered our WAFT-related novelty claims. The contribution statement now says we "adapt" warping-based feature sampling to 3D rather than "introduce" 3D WAFT, and the abstract, introduction and conclusion no longer list it among our key innovations. It nevertheless remains a critical design choice: it is the matching operator that lets us refine all points at once and scale to all-frame tracking.

With regard to making a comparison using the same input geometry and query points, Table 1 actually already provides this. TAPIP3D is given the same VGGT-Ω pointmaps and the same TAPVid-3D query points as TrackEverything. TrackEverything is on par or better (APD-P 34.7 vs. 34.1 on the first 48 frames, 31.2 vs. 29.8 on full-length videos) while also tracking every other point in the video. Within our model, removing 3D WAFT lowers APD-P from 31.2 to 29.3 (Table 2). CoWTracker is a 2D tracker without geometry input, so TAPIP3D is the matched-input comparison.

When an object rotates, newly visible surfaces can contribute new points. What happens to points that were tracked before becoming hidden? If they reappear, how does the method continue their original tracks instead of creating new ones or merging them with nearby points?

Great question. The key is that hidden points are not dropped, they persist through the occlusion and receive position & visibility updates just like visible points. The model is trained to track through occlusion. Then, when the surface reappears, it gives birth to a new copy, and de-duplication merges the two copies if they agree at their predicted end-of-window positions.

TrackEverything uses estimated geometry to place points in 3D and visual features to refine their motion. (1) If a pointmap or camera pose is wrong, can visual evidence correct the correspondence before a false voxel merge occurs? (2) Could establishing correspondence primarily from features, then using geometry for 3D placement and merging, reduce these errors? Results grouped by geometry error would clarify how the two signals interact in TrackEverything.

Thank you for these questions. To separate the roles of geometry and correspondence in wrong merges, we measure 'harmful' merges i.e. merged pairs whose ground-truth motion later diverges by more than 0.04 units. We study four settings: (a) Our model with GT geometry; the harmful merges here are due only to wrong correspondence (b) Our model with estimated geometry (c) Perfect 2D correspondence (GT 2D tracks) lifted with the estimated geometry; here merge errors are due only to geometry. (d) Same as c but but with learned 2D correspondences using CoTracker3’s 2D tracks. We run these tests on PointOdyssey (POD) and Dynamic Replica (DR) where we have near-dense GT tracks. We run it on 128 frame long videos (7 window merges) and also on 32 frame windows. Tracking every pixel of every frame with CoTracker3 is expensive (~400k query points for a 32-frame video, 20 mins/video), so we run (d) on 32-frame videos only.

Harmful merges (% of merged pairs, lower is better)

SettingCorrespondenceGeometryPOD, 32 fDR, 32 fPOD, 128 fDR, 128 f
(a)OursGT0.01%0.00%0.20%0.06%
(b)OursEstimated (VGGT-Ω)0.11%0.06%0.72%0.18%
(c)Perfect 2D (GT tracks)Estimated (VGGT-Ω)0.12%0.03%0.36%0.06%
(d)CoTracker3 2D tracksEstimated (VGGT-Ω)0.20%0.03%––

32 f: 32 frames. 128 f: 128 frames

Here are our conclusions:

(i) Harmful merges are rare even in the full setting with estimated geometry: 0.11% / 0.06% (POD / DR) on 32-frame videos, and 0.72% / 0.18% on 128-frame videos.

(ii) Most of them are caused by geometry errors. With GT geometry, harmful merges drop to 0.01% / 0.00% (32 frames) and 0.20% / 0.06% (128 frames). Errors in the input geometry therefore account for roughly 70% or more of the harmful merges. Even with perfect correspondence, estimated geometry still yields more harmful merges than our model with GT geometry on POD (0.36% vs. 0.20%).

(iii) Perfect correspondence recovers part of the error, but learned 2D correspondence gives no consistent gain. Perfect 2D tracks halve harmful merges on 128-frame videos (0.72% → 0.36% POD, 0.18% → 0.06% DR). CoTracker3's learned 2D correspondences nearly double harmful merges on POD (0.20% vs. 0.11%) and halve them on DR (0.03% vs. 0.06%), while keeping about 1.5× more tokens on both (thus failing to optimally de-duplicate).

We added this analysis in Sec. A.8 and Table 5b of the supplementary (in blue).

What appearance information does the merged track retain, and which feature serves as the source feature for 3D WAFT in later windows? Does matching become less accurate after a large viewpoint change or after several merges?

We mean-pool the appearance features of the merged points, and this pooled feature serves as the 3D WAFT source feature in later windows. Like all trackers, ours gradually loses accuracy with video length and viewpoint change, but merging adds little further degradation. On full-length videos in Table 1, TrackEverything remains competitive with state-of-the-art sparse trackers that perform no merging, and shows similar degradation in performance when going from 48 frames to 128 frames. We also measured merge quality directly on 128-frame videos: the fraction of harmful merges grows only slowly with the number of window boundaries, from 0.21% to 0.72% of point pairs on PointOdyssey and from 0.08% to 0.18% on Dynamic Replica over seven boundaries (Supp. Sec. A.8, Table 5a).

In Table 3, replacing estimated Pi3 geometry with ground-truth geometry raises TrackEverything’s APD-P from 36.6 to 85.2 on Dynamic Replica and from 24.9 to 73.2 on PointOdyssey. DeltaV2 also improves, but by a smaller margin. What explains the larger change for TrackEverything?

While it is hard to pinpoint the exact mechanism, we believe the difference lies in coordinate system differences. TrackEverything attends over a world-coordinate 3D feature cloud, so it can directly benefit from more accurate point clouds and camera poses. DeltaV2 operates in camera coordinates, so it cannot fully exploit accurate camera poses even when they are available.

Figure 3 compares runtime and memory on an L40S GPU, but the methods may track different numbers of points. For each method, does the reported runtime include geometry prediction, and how many points are tracked over the video, including points added after the first frame? Reporting these numbers would make the speed comparison easier to interpret.

Thank you for pointing out these ambiguities.

Geometry. In the submitted version, Fig. 3 reported tracking cost only. We have updated Fig. 3 and Fig. 11 to also include the inference time and memory of geometry estimation. All trends remain similar, and the original tracking-only figure is now Fig. 12.

Number of points. Fig. 3d–f controls for this: the x-axis is the number of query points (100–10k), matched across methods. The baselines track exactly these queries. TrackEverything tracks the same queries plus every point in every frame, including points that appear after the first frame, since its dense tracking cannot be switched off. Even so, its latency and memory stay low and grow slowly with the number of queries. For reference, a 400-frame PointOdyssey video holds ~70k persistent tracks after de-duplication, versus 4.9M if every pixel of every frame were tracked.

In the supplementary figures and the project website, TrackEverything and the baselines visualize different sets of points. The captions also state that only dynamic trajectories are shown. How were the points and trajectories selected for each method? Could the authors show all methods tracking the same points from the first frame, with identical visibility and rendering settings, so that correspondence quality can be compared directly?

Point selection. Our original visualizations were designed to highlight a new capability: TrackEverything efficiently tracks all points across all frames of a video, rather than only first-frame points or a sparse set of queries. It therefore needs no point sampling. For the baselines, we sampled query points on a grid in the first frame. Denser grids, as well as grids sampled on every frame, ran out of memory.

Same points, same rendering. Following your suggestion, we added a controlled comparison (Rebuttal section of https://iclr-8302.netlify.app/ and Fig. 8 in the appendix). Every method tracks the same 40×40 grid sampled on the first frame, and all tracks are drawn with the same renderer. DeltaV2, CoTracker3, SpatialTracker-v2 and TrackEverything all perform well, with slightly cleaner tracks for TrackEverything, while VDPM struggles noticeably. This matches Table 1, where TrackEverything is competitive with sparse and first-frame dense trackers and significantly outperforms VDPM. The key advantage of TrackEverything is that it delivers this accuracy while tracking every point across long videos.

Thank you again for a thoughtful and detailed review that has made the paper stronger. We believe the revision addresses your main concerns, in particular the new analysis separating geometry and correspondence errors in de-duplication (Sec. A.8), the matched same-points comparison (Fig. 8 and website), the tempered 3D WAFT claims, and the experiment with VGGT-Ω features. If so, we would be grateful if you would consider raising your score, or let us know what concerns remain.

Reviewer_KugEOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents TrackEverything, a 3D point tracking framework for dense long-horizon video tracking. The method maintains a persistent world-coordinate scene representation and processes videos using sliding windows. At each window, an endpoint refiner estimates the boundary positions and static/dynamic labels of scene points, while a trajectory refiner predicts dense trajectories for dynamic points. A voxel-based de-duplication mechanism merges repeated observations across windows, and 3D WAFT replaces memory-intensive 4D correlation volumes. Experiments on several benchmarks show strong tracking accuracy, favorable memory usage, and the ability to process long videos.

Critical strengths
  1. The paper addresses an important limitation of existing dense 3D trackers: their memory and computation grow with video length. The persistent world-coordinate representation and cross-window de-duplication provide a clear solution to this problem.
  2. The experimental results are strong and consistent across several benchmarks. The ablation study is also comprehensive and helps clarify the contribution of the proposed components.
  3. The paper is clearly written and logically organized, making the proposed method and experimental findings easy to follow.
Critical weaknesses
  1. The long-horizon advantage is closely tied to the proposed sliding-window inference and cross-window feature concatenation. It is unclear how much of the improvement comes from the persistent 3D representation itself and how much comes from the inference protocol. In particular, it would be useful to clarify whether a comparable windowed strategy could also be applied to the dense baselines.

  2. Cross-window de-duplication assumes that static points from adjacent windows are accurately aligned in the shared world coordinate system. If the predicted positions have noticeable errors, the same surface may fail to merge, while nearby surfaces may be incorrectly merged through mean pooling. Since this operation is irreversible, the paper should discuss (or quantify) its effect on long-horizon accuracy and track identity.

Questions for authors
  1. Could the same non-overlapping window inference and state concatenation strategy be applied to the two dense baselines? If not, what specific architectural or computational limitation prevents this comparison?
  2. How sensitive are the results to cross-window alignment errors in static regions? For example, what happens when the position error between two observations of the same surface is larger than the voxel size?
  3. If most of the scene is dynamic and only a small fraction is static, does this affect the tracking reliability? In particular, could dynamic points accumulate across windows or affect the initialization of later tracking? If possible, the author can discuss this case or provide a simple stress test.
2 comments

Could the same non-overlapping window inference and state concatenation strategy be applied to the two dense baselines? If not, what specific architectural or computational limitation prevents this comparison? It is unclear how much of the improvement comes from the persistent 3D representation itself and how much comes from the inference protocol.

Great question. First of all, persistent de-duplicated 3D representation itself is critical to scale to long-videos. In Table-2 (d), we run our method without de-duplication ("no vox") and find that it quickly goes OOM. The windowed protocol alone is not enough.

The dense baselines show the same thing. DeltaV2 already uses sliding windows and still runs out of memory at ~200 frames while only densely tracking first-frame pixels (Fig. 3). VDPM, D4RT and Any4D cannot use such a protocol at all: they map video frames directly to tracks and do not take a previous window's track as input to continue their tracks forward.

Carrying state efficiently across windows needs both a recurrent design that accepts and refines previous estimates and a representation that does not grow with frame count. Our endpoint refiner and voxel de-duplication provide exactly these two functions.

Cross-window de-duplication assumes that static points from adjacent windows are accurately aligned in the shared world coordinate system. If the predicted positions have noticeable errors, the same surface may fail to merge, while nearby surfaces may be incorrectly merged through mean pooling. Since this operation is irreversible, the paper should discuss (or quantify) its effect on long-horizon accuracy and track identity.

Thank you for the suggestion. We have quantified this now and added it to the supplementary. For each merge by our model, we check the ground-truth for (i) whether the two points were on different surfaces at the merge time (i.e. were they more than two voxels away in their ground-truth locations?) (ii) whether their GT motions later separate by more than two voxels (0.04 units). Only (ii) actually harms the output tracks, since merged points incorrectly share one motion from then on. We run this analysis on PointOdyssey (POD) and Dynamic Replica (DR) with near-dense GT available, and we use estimated geometry from VGGT-Ω for our model. We find that 3.6% / 2.4% (POD / DR) of merged pairs merge points more than 2 voxels apart. The rate of harmful merges, where the two points later move apart, is only 0.72% / 0.18% (POD / DR).

How sensitive are the results to cross-window alignment errors in static regions? For example, what happens when the position error between two observations of the same surface is larger than the voxel size?

In this case, the two observations are not merged: both are tracked forward as separate (and individually correct) tracks, and may merge at a later window boundary. A missed merge therefore costs compute without affecting the accuracy. Table 2(d) quantifies this: shrinking the voxel from 0.02 to 0.005 causes many more missed merges, which raises memory usage from 15 to 24GB, while keeping the accuracy nearly unchanged (30.9 to 31.4).

If most of the scene is dynamic and only a small fraction is static, does this affect the tracking reliability? In particular, could dynamic points accumulate across windows or affect the initialization of later tracking? If possible, the author can discuss this case or provide a simple stress test.

Dynamic points do not accumulate across windows: they are also merged at the end of the window by using their predicted 3D locations at the end time-step of the window. Our merge rule only requires that points occupy the same voxel at the same time, so it applies to moving points too. Table 2(b) also gives a stress test of the extreme case: removing the static/dynamic classifier treats every point as dynamic and decodes its full trajectory. Accuracy is unchanged (31.2 APD-P). Only the inference cost rises (0.05 → 0.24 s/frame, 9.1 → 22.4 GB). A mostly dynamic scene therefore costs extra compute for trajectory decoding but does not reduce reliability. Dynamic-heavy examples are in the long-video section of our website (https://iclr-8302.netlify.app/).

Thank you for your careful review and for your strong support for our work. We would be happy to clarify any remaining questions you may have.

I thank the authors for the detailed responses and additional analysis. The clarification of the dense baselines’ limitations, together with the ablation without de-duplication, strengthens the evidence that long-horizon scalability depends on the proposed representation beyond sliding-window inference alone. The merge-error analysis also helps address my concern about irreversible merging, with low rates of harmful merges on the two evaluated datasets. I therefore maintain my original rating of clear acceptance.

Reviewer_EXJgOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The submission offers an approach that allows 3D point tracking of dynamic + static elements in VERY long monocular video sequences, e.g. over 1000 frames long. The authors propose a solution that scales really well with the number of frames and the number of tracked points. The "secret" is to use VGGT-Omega plus examine static vs dynamic 3D space occupancy of voxels in one 16-frame block at a time, and then do a careful re-set for the next 16-frame non-overlapping block-boundary. A reset has all track IDs that land in the same voxel at the end of a block being mean-pooled into one track.

Critical strengths

It's truly impressive to cope with motion of general dynamic shapes for 1000 frames! Yes, the error accumulates because of the proposed approach, but it's better than running out of memory, or reconstructing just the static parts.

I like the endpoint-refiner and trajectory refiner for being sensible: dense yet bounded. The endpoint-refiner does while the trajectory refiner does not take the entire context into account (correlation with other tracked points).

Minor, but it's also nice to see corrections/improvements to the PointOdyssey dataset.

Critical weaknesses

• The results on Pi3 geometry come out at or below every baseline, within 0.1 points of the weakest. So the proposed method's success hinges entirely on VGGT-Ω's success for a given scene. How was VGGT-Ω run on videos with 100's or even 1000 frames? VGGT-Long and related papers are really pertinent to this task, but there's no mention here.

• Related, the claims about efficiency + handling of long sequences assume (acknowledged cleary, including in Limitations) that some other mechanism has estimated geometry (poses and depths). Where is the memory-budget for that processing counted? (Else it's trivial to claim "we're memory efficient as long as we don't count the memory-inefficient parts".) TAPIP-3D tracks in windows of 16 frames too (in a CoTracker3 kind of way). It does depend on external depth (not poses for the camera-centric mode) but I don't think it made claims about memory-efficiency and handling of long sequences. Why is it excluded from Fig's 3 + 10 please?

• Reference to [Point3R NeurIPS 2025] is missing. Only for static points it showed how to do streaming reconstruction of points in 1000 frame sequences using 3D-anchored memory tokens and mean-pooling. The proposed method maintains track ID's which unlocks dynamic object tracking, so there is still added value here, just need to acknowledge which part is actually new.

• There's no reference to [Voxel Hashing, Siggraph 2013] or to [NeuralRecon, CVPR 2021] or [TransformerFusion, NeurIPS 2021] all of which fused learned features in a sparse or voxel representation. Again, those were for static scenes, but de-duplication itself isn't new.

• 3D WAFT is claimed as novel and implied to be a new way to avoid correlation volumes. The novelty of WAFT 3D is tempered by the fact that CoWTracker CVPR 2026 had imported WAFT to dense 2D point tracking, in much the same way. They listed depth-aware warping as possible future work. That's not the same as doing it, but (here) given depths, it feels like a fairly straightfoward "lift". On that backprojection to the camera(s), is care taken about the many possible views of that geometry? And perhaps a minor point, but the acronym WAFT in the Wang + Deng paper stood for Warping-Alone Field Transforms, but the authors here are referring to their WAFT 3D as Warp-Aligned Feature Transform, which is confusing.

• The static vs dynamic determination: what happens with points that are stationary for a while, but begin to move mid-block? And does anything correct errors across windows?

• Another reference to add could be [Shape of Motion, ICCV 2025] which isn't a single GPU feed-forward pass, but does reconstruct the full dynamics of a video as a "globally consistent representation of the dynamic scene". They had a similar premise, that if other models can provide poses, depth, and even 2D tracks, then they could reconcile everything by reasoning in 3D. Where the proposed method refines trajectories, they refined them using motion bases.

Questions for authors
  1. How were outputs shifted + scaled into the camera's frames for scoring?
  2. What geometry was used for training? Ground truth?
  3. Are the authors aware of TrackingWorld from NeurIPS 2025? It is set up to run only for short clips, but it tracks almost all pixels in world coordinates, separates static from dynamic points, and even de-duplicates, discarding new points that land near existing tracked points. Yes, the claims of "first" here refer to the ability to cope with 1000 frame sequences, but that emerges from doing a reset after every 16 frame block, and much of the work within a block has cite'able predecessors.
2 comments

Thank you for your review. Changes in the revised manuscript are marked in blue.

How was VGGT-Ω run on videos with 100's or even 1000 frames? VGGT-Long and related papers are really pertinent to this task, but there's no mention here.

We run VGGT-Ω on 60 frame chunks, with 30 frame overlap, and do a small Sim(3) alignment between the chunks. This is the VGGT-Long strategy (Deng et al.) minus loop closure, which we omitted for efficiency. We missed citing this important work; thank you for pointing it out. We now cite it and describe this procedure in the supplementary (Sec. A.1, in blue).

The results on Pi3 geometry come out at or below every baseline, within 0.1 points of the weakest. So the proposed method's success hinges entirely on VGGT-Ω's success for a given scene.

We agree that TrackEverything's Pi3-based results are within 0.1 points of the weakest sparse point tracking baseline (not all baselines), but under Pi3, all sparse tracking methods are within 1 point with each other. We only claim to be competitive with sparse trackers, which holds up, on both Pi3 and VGGT-omega. Our main success is with respect to all-point dense tracking. There, even with Pi3, we score about 2.8x the APD-P of VDPM (30.2 vs 10.9) and we are the only all-frame dense method that runs on full-length videos.

Why memory-budget for geometry estimation is excluded from Fig 3 + Fig 10? Maybe geometry estimation is the memory-inefficient part?

Thank you for raising this. We updated Fig 3 and 10 to include the inference and memory cost of geometry estimation as well. All trends still remain similar to our submitted version; we include the original figure with tracking only costs as Fig 11 for comparison. Notably, geometry estimation does not induce a memory bottleneck. It always runs on chunks of 60 frames and does not accumulate scene content the way all-point tracking must, so its peak GPU memory stays nearly flat: 11.4 GB at 120 frames vs. 14.2 GB at 1000 frames. The inefficiency of the baselines come from tracking itself, where they still run OOM irrespective of GT geometry or estimated geometry.

Reference to [Point3R NeurIPS 2025], Voxel Hashing, NeuralRecon, Transformerfusion; while these methods operate only for static scenes, but de-duplication is not new.

Thank you for pointing out these references. Indeed they are very relevant and we now include them in our related work. As you point out, most works which use de-duplication only target static scenes. We extend de-duplication to dynamic scenes with a simple observation: two tracked points that occupy the same voxel at the same time are duplicates of the same surface. Our endpoint refiner therefore first predicts every tracked point's location at the last frame of the window, and we de-duplicate all points at that shared time step. For a more detailed discussion, please check the newly added 'Long-Video 3D Reconstruction' paragraph in Related Works.

3D WAFT is claimed as novel to avoid correlation volumes; but novelty is tempered since CoWTracker used it for 2D dense tracking. WAFT's acronym has a typo.

We agree. We now have tempered the novelty claims of 3D WAFT in our writing. The contribution statement now says we "adapt" warping-based feature sampling to 3D rather than "introduce" 3D WAFT, the abstract, introduction and conclusion no longer list it among our key innovations We also fixed the mistake in WAFT's acronym.

Is care taken about the many possible views of that geometry during WAFT?

Yes. We perform WAFT sampling independently for each point in the window. Even when the geometry holds several copies of the same point (e.g., a static point visible at multiple timesteps), each copy is lifted to 3D on its own, projected into the target frame, and samples that frame's image features. Each copy therefore has a single, well-defined projection: we unproject it from its source view using depth and intrinsics, move it to the target view using the camera extrinsics, and project it with the target intrinsics.

The static vs dynamic determination: what happens with points that are stationary for a while, but begin to move mid-block?

The classification will shift from static to dynamic in such a case; This is because the static/dynamic classification happens per sliding window depending on whether the model thinks if the object moved in that window or not. If an object moves in any part of the window, it is classified as dynamic, and we decode full-length T tracks for it in that window (irrespective of its motion classification in prior windows). Some good examples of this is the 6th video of "Static/Dynamic Classification" section on our website (https://iclr-8302.netlify.app/) with three cats, where you can see labels changing on the cat when it moves; or the 10th video from PointOdyssey where objects shift labels between static/dynamic across windows.

And does anything correct errors across windows?

Yes. The endpoint carried over from the previous window is only an initialization. In the next window, the endpoint refiner is free to move it, with exactly the same mechanism as its within-window iterations (Sec. 3.3). The source feature is always sampled at the point's birth location, never at a previous estimate, so in every window the model compares the current target appearance against the point's original appearance. A point that drifted in one window therefore produces an appearance mismatch in the next one, which the refiner is trained to correct: training supervises offsets toward the GT location for all points, including those carried over from earlier windows

Add reference to Shape of Motion

Thank you, we now cite Shape of Motion in the related work.

How were outputs shifted + scaled into the camera's frames for scoring?

We follow the official TAPVid-3D evaluation protocol, identically for all methods. Our tracks are predicted in the world frame of the estimated geometry, and we rigidly transform them into each frame's camera coordinates using the estimated camera extrinsics from the same geometry backend (e.g., VGGT-Ω). We apply the standard TAPVid-3D global median scaling: a single scale factor per video, the ratio of the median norms of GT and predicted points over points visible in both.

What geometry was used for training? Ground truth?

Yes, ground truth.

Are authors aware of TrackingWorld? It tracks all pixels in short clips, seperates static-dynamic points, de-duplicates its tracks; Claims of "first" refers to the ability to tracks 1000+ frame sequences, but that emerges from doing a reset after every 16 frame block, and much of the work within a block has cite'able predecessors.

Thank you for pointing us to TrackingWorld; we were not aware of it and now cite and discuss it in the related work. TrackingWorld obtains tracks for almost all pixels by running a first-frame dense 2D tracker from every frame, discards tracks that lie near already visible 2D tracks, segments dynamic objects with a VLM and Grounding-SAM, and finally lifts the 2D tracks to world-centric 3D through per-video bundle adjustment and as-rigid-as-possible optimization. This takes ~20 minutes for a 30-frame video. Similar to them, TrackEverything also does de-duplication and static/dynamic classification. Differently from them, TrackEverything does static/dynamic classification and de-duplication natively within the architecture, runs at around 7 FPS including geometry estimation (23 FPS tracking only) and scales to 1000+ frame videos.

Thank you again for a careful review that has made the paper more accurate. We believe the revision addresses your main concerns, in particular the geometry cost (now counted in Figs. 3 and 11) and the missing related work. If so, we would be grateful if you would consider raising your score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Author response summary

We thank all reviewers for their thoughtful reviews and constructive suggestions. We are pleased to receive numerous positive remarks, such as:

  • [EXJg]: "truly impressive to cope with motion of general dynamic shapes for 1000 frames," "sensible: dense yet bounded."
  • [KugE]: "addresses an important limitation of existing dense 3D trackers," "a clear solution to this problem," "experimental results are strong and consistent across several benchmarks," "clearly written and logically organized."
  • [3SSE]: "tackles a meaningful scaling problem," "all-frame coverage can be maintained at a computational cost that was impractical for the evaluated all-frame dense baselines," "the ablations support the efficiency benefit of this decomposition."

Following the reviewers' suggestions, we made the following revisions, all highlighted in blue:

  • Include Geometry estimation cost: runtime and memory charts now include geometry estimation cost (Fig. 3, Fig. 11); all conclusions hold.
  • False-merge analysis: we measure how often de-duplication merges the wrong points, and whether a correspondence-first alternative reduces such errors. Harmful merges are rare, and the alternatives are costly and does not consistently reduce errors.
  • Geometry-backbone features: replacing DINOv3 with VGGT-Ω features gives on-par accuracy.
  • Same-point tracking visualization: a visualization where all methods track the same points with identical rendering (Supp. Fig. 8, website).
  • Additional Related work and tempered novelty claims for 3D WAFT: we added Point3R, VGGT-Long, Voxel Hashing, NeuralRecon, TransformerFusion, Shape of Motion and TrackingWorld, and tempered the novelty claims for 3D WAFT.

We would like to reiterate our key contribution: TrackEverything is the first feed-forward method to track every point across every frame of 1000+ frame videos. It does this by representing a video as a persistent 3D scene whose cost grows with unique scene content, not with video length. We believe this capability will open up new applications involving VLMs with persistent state, long-horizon video generation, and robotics, making it broadly useful to the community.

Please see below for our responses to each reviewer's questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.