acceptodds
Under review as a conference paper at ICLR 2027

TrackDiT: 3D Tracking from 2D Tracks and 3D reconstruction

Abstract

Recent 3D point trackers adopt architectures that jointly learn appearance matching, 3D reconstruction and temporal reasoning in a single yet complex end-to-end model. By evaluating a simple baseline that lifts 2D tracks into 3D using an off-the-shelf 2D tracker and 3D reconstructor, we derive two insights: (i) such a baseline outperforms specialized 3D tracking models on always-visible tracks, and (ii) when tracks are partially occluded, specialized models surpass such a baseline. This observation motivates a new paradigm: rather than training one network for everything, we first obtain precise visible track fragments from foundation models, then treat the remaining occlusion gaps as a track completion problem, addressed by a flow-matching model that generates the correction to the lifted trajectory. Because a hidden point’s motion is genuinely ambiguous, a generative formulation is a better fit than regression, and operating on track-frame tokens lets the model complete a gap both from the track’s own motion and from the co-visible points around it. The resulting framework is modular, simple to train, and automatically improves as its foundation-model components advance. TrackDiT achieves state-of-the-art 3D tracking results on ADT, PStudio, and SynthVerse. In particular, on occluded points it improves over the lifting baseline by 12.2 APD3D points and surpasses specialized models, while also improving accuracy on visible points.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
point tracking, 3d tracking, flow matching, diffusion transformer, dynamic 3d reconstruction

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_B58LOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The following is a brief summary of the TrackDiT paper as understood by this reviewer:

  • The authors show that 3D trackers are not good at transferring their tracks back to the 2D image, despite claims of good 3D tracking and camera pose estimation.
  • They show a simple baseline of 2D tracking with 3D lifting out-competes these 3D trackers on visible points but struggles when points undergo occlusions due to other objects and out-of-frame.
  • The authors introduce a new modular method which utilises the best of 2D trackers and 3D trackers to lift 2D tracks to 3D before using a flow matching design to de-noise tracks in 3D so that they perform better across occlusion gaps.
  • They show consistent improvement in 3D tracking over prior state-of-the-art models.
Critical strengths

The following is a list of strengths for the paper:

  • The paper is well presented and easy to understand. The authors start with a thorough motivation for their work by going through the pros and cons of 2D and 3D trackers, with a comparison to a simple baseline. The method then follows on nicely from this start before results are presented that back up their claims well.
  • The authors include interesting and informative diagnostics of point tracking performance split by visible and occluded points. This is nice to see as it points out that they are very different challenges and further supports the gap that their work looks to fill.
  • It is good to see the comparison of 2D and 3D tracking. Some works (e.g. SpatialTracker) have made the claim that tracking is easier to conduct in the 3D space, but not many have thought about how this translates back to the 2D plane once we have good 3D tracks and whether these models are even very good at this new task.
  • The method section is well motivated, supported by simple and effective figures as well as being a fairly elegant solution to the problem.
  • The authors show state-of-the-art performance across a number of benchmarks and against a large number of 3D trackers.
Critical weaknesses

The following is a list of things that this reviewer considers weaknesses of this paper:

  • It would be more complete to see a simpler baseline of simple interpolation to fill the occluded regions in 3D. It is likely that this will be much worse but it will help the reader understand the kind of motion that is found in the benchmarks and whether the diffusion process is learning to predict any complex motion.
  • A more detailed discussion on the process of handling out-of-frame points would be good to see. Do 4RC, Any4D and TraceAnything already handle this? Or are the authors assuming something for these models when points leave the frame?
  • Although the main table shows good improvements against 3D trackers, it would also be good to see a selection of 2D trackers in this evaluation. This is not a strong weakness, but more of an interesting point to make the paper more complete.
  • In the implementation details, the authors mention training on 48-frame windows. However, there does not seem to be any analysis of point tracking accuracy over regions shorted and longer than this training window. This seems like an important ablation to include. The fact that large residuals are left out of training potentially points to the method maybe struggling quite a lot in long-term occlusions, especially out of frame.
  • The weighting of the loss function is understandable but the paper is missing a proper ablation on how these weightings were chosen. For example, why are occluded and out-of-frame weighted the same? As surely out-of-frame is harder than simply occluded within the frame?
  • Although the ablations are good, they do appear to be quite limited. For example, there are no ablation on the number of layers, dimension sizes, number of Euler steps, region weightings, training window size etc.
  • The qualitative examples look really nice. However, there does not seem to be many with tracks leaving the frame or being occluded through complex motion. If these were included, then they would be more convincing.
Questions for authors
  1. How are out-of-frame points handled by the baselines?
  2. Is simple interpolation between visible points evaluated as a baseline anywhere?
  3. Why are occluded and out-of-frame weighted the same in the loss?
  4. How does tracking accuracy fair when the model needs to predict large residuals for refinement in the flow matching stage? E.g. if the actual motion is very complex and far from the noise/cold start then does it struggle?
2 comments
AuthorsOpenReview ↗

Response to Reviewer B58L

We thank the reviewer for the positive assessment and the many concrete suggestions. We ran all requested analyses and ablations.

Part 1

W1 / Q2. Simple interpolation baseline. See G1.

  • Interpolation alone helps: linear or spline interpolation and the per-track median fill reach 55.3-55.8.
  • TrackDiT is clearly above all of them: +2.5 overall, +5.0 on occluded points.
  • The model learns real motion: its largest gains come exactly where interpolation has the least to work with, namely out-of-frame points and large errors (see below).

W2 / Q1. How the baselines handle out-of-frame points. No assumption is made for any baseline; each outputs a 3D point for every query in every frame:

  • 4RC, V-DPM, Any4D and TraceAnything predict per-pixel trajectories (or points plus scene flow) from frame 0. We read every frame at the query's frame-0 pixel, so out-of-frame frames come directly from the model.
  • TAPIP3D, SpatialTrackerV2 and DELTA track each query and output a 3D position and a visibility for every frame.
  • The Simple Baseline replicates the border of the depth map (footnote 1).

Accuracy per point class, with each method under its own alignment:

MethodAllVisibleOccluded in frameOut of frame
TrackDiT58.3862.7746.8047.47
Lift + per-track median55.9460.7742.3339.09
SpatialTrackerV2 (VGGT-Ω geometry)53.7558.3342.4040.64
TraceAnything42.6546.0534.5732.91

TrackDiT is best in every class, and by the widest margin out of frame: +8.4 [+6.3, +10.9] over the lift and +6.8 [+4.7, +9.1] over SpatialTrackerV2.

W3. 2D trackers in the evaluation. We added three rows, each a 2D tracker followed by our lift:

2D tracker+ lift (9-col)+ TrackDiT (9-col)
CoWTracker55.8258.33
CoTracker355.1257.01
SpatialTrackerV2 (2D tracks)51.6654.87
  • TrackDiT never saw CoTracker3 or SpatialTrackerV2 in training, and improves both.
  • With SpatialTrackerV2's 2D tracks, TrackDiT even surpasses SpatialTrackerV2's own 3D output (53.93).

Part 2

W4. Shorter vs. longer than the 48-frame training window; long occlusions. TrackDiT's gain over its input, grouped by how long a point stays hidden:

Hidden run1-4 frames5-1617-48> 48 (beyond the window)
TrackDiT - input+5.86 [+4.86, +6.88]+5.31 [+4.29, +6.29]+5.86 [+4.05, +7.61]+5.42 [+2.86, +7.51]

The gain stays the same for occlusions longer than the training window. We also retrained with 24- and 64-frame windows (ablation recipe, all 11 subsets). Both stay within 0.4 points of the 48-frame model (24 frames: +0.32 [+0.04, +0.62]; 64 frames: +0.18 [-0.15, +0.49]), so the method is insensitive to the window length.

W5 / Q3. Region weights; occluded vs. out-of-frame. Ablation recipe, all 11 subsets:

vis : occ : out-of-frameAllOccluded
1 : 1 : 161.25 (-0.85 [-1.17, -0.54])53.59
1 : 3 : 3 (paper)62.0954.70
1 : 3 : 662.23 (+0.14 [-0.13, +0.42])54.74
  • Up-weighting hidden points is important: removing the weighting costs 0.85 overall and 1.1 on occluded points.
  • A higher out-of-frame weight does not help: it gives no significant change, so the simpler equal weighting is enough.

W6. More ablations. Ablation recipe (all 11 subsets), and the number of Euler steps on the main model (9-col):

VariantAllvs. paper setting
Depth 12, width 512 (paper)62.09-
Depth 661.12-0.97 [-1.34, -0.63]
Depth 1860.69-1.40 [-1.71, -1.10]
Width 38461.15-0.94 [-1.29, -0.63]
Width 768 (2.25x parameters)62.22+0.13 [-0.30, +0.49]
Euler steps1241020 (paper)60
9-col56.2158.3058.1858.3558.2858.07
  • Model size: the paper's configuration is the best depth, and width saturates at 512.
  • Euler steps: accuracy is similar between 2 and 60 steps; a single step is clearly worse.
  • Training window: see W4.

W7. Qualitative examples. We refer the reviewer to the supplementary videos, which cover both kinds of occlusion under complex motion:

  • Occlusion by other objects: swing, bmx-trees (the cyclist passing behind the tree trunks) and e-bike.
  • Self-occlusion of a moving body: parkour, tennis-vest and rollerblade, where limbs repeatedly occlude each other and the person is occluded by objects.

In all of them, the traces stay smooth through the occlusions, without the flicker or flying points of the raw lift, which shows that the model reasons correctly about the hidden motion. In the revision we will add more scenes and viewpoints, focusing on points that leave the frame and on long occlusions.

Q4. Large residuals. Tokens grouped by the lift's error before refinement:

Lift error2nd quartile3rd quartileLargest quartileLargest quartile, hidden points
TrackDiT gain+2.20+3.83+9.59 [+8.13, +11.04]+12.35 [+9.70, +14.10]

The model does not struggle when large corrections are needed; its gain grows with the size of the required correction, and is largest (+12.4) for hidden points with the largest lift errors, which is exactly the case the reviewer asked about.

Reviewer_eJ72OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper does 3D point tracking in two steps. First, it lifts 2D tracks from an off-the-shelf 2D tracker (CoWTracker) into 3D, using depth and cameras from a 3D reconstruction model (DA3 or VGGT-Ω). The authors show this simple baseline matches or beats end-to-end 3D trackers on visible points, but fails on occluded points, where the depth map gives the depth of the occluder. Second, to fix this, they train a small transformer (TrackDiT, ~80M parameters) with flow matching to predict a correction to the lifted 3D tracks. It uses only geometric inputs, with one token per query point per frame. On ADT, PStudio and SynthVerse, TrackDiT gets the best average score. Most of the gain is on occluded points.

Critical strengths
  1. Clear fix for a clear problem. The paper shows exactly where 2D-to-3D lifting fails (occlusion), and the method targets that failure. As the model only predicts a correction, it achieves real gains for occluded points, and visible points do not get worse. This shows that a small, geometry-only module on top of frozen models can compete with end-to-end 3D trackers.

  2. Simple and modular. The base models are frozen and the correction model uses no image features, so the method is easy to understand.

Critical weaknesses
  1. Limited novelty, and the use of a generative model is not justified. Since the analysis in Sec. 3 is mostly expected, the contribution rests on the correction model. The paper says a generative model is needed because "a hidden point's motion is genuinely ambiguous" (abstract, l107, l287). But this is never tested:

    • There is no comparison with a plain regression model of the same architecture (Tab. 4 does not include one).
    • Results use a single sample from random noise. There are no results over several seeds, and no mean-of-K or best-of-K results. For a distance-based metric, one random sample should not beat the average prediction.
    • Sec. 4.5 says co-visible points on the same object show the hidden point's motion directly (l399–402). Spatial attention is also the most important part in Tab. 4. This suggests the gains come from clear cues from other tracks, not from modeling ambiguity.

    Without these tests, the method reads as an engineering trick (residual correction + factorized attention) on top of strong frozen models. This is the main reason for my rating.

  2. Speed and scaling are claimed but not measured. The paper calls the model "small, fast" (l297) but reports no runtime or memory, neither for the correction model nor end to end. Each token is one query point at one frame, and inference takes 20 denoising steps. So cost grows with the number of query points, and spatial attention grows quadratically with the number of tracks. CoWTracker itself gives dense tracks, but the correction model cannot easily handle dense query sets. The model is trained with only 128 tracks, and the paper does not say how many tracks are processed together at test time. Being simple and cheap is a key selling point over end-to-end models, so this needs evidence.

  3. The evaluation may favor TrackDiT. TrackDiT is trained on SynthVerse, which also makes up 7 of the 9 test columns. According to their papers, 4RC and V-DPM were not trained on SynthVerse, so they are tested zero-shot there. Their other dynamic training data overlaps with TrackDiT's (PointOdyssey, Kubric, and Dynamic Replica for 4RC), so SynthVerse is the main difference.

Questions for authors
  1. Could author add comparison with the same model trained as a direct regressor, and also report mean + std over several seeds, and mean-of-K / best-of-K results.
  2. Please report runtime and memory end to end (base models + correction model) against 4RC, V-DPM and SpatialTrackerV2. How do accuracy and cost change with the number of query points (e.g., 128 / 1k / 10k)? How many tracks are processed together at test time? How would dense tracking work?
  3. How does TrackDiT compare to simple fixes for occluded points? For example, keep the 2D tracker's ray and interpolate the depth from nearby visible frames.
2 comments
AuthorsOpenReview ↗

Response to Reviewer eJ72

We thank the reviewer for the precise and actionable review. We ran every requested experiment. In particular, they support the generative formulation: flow matching outperforms the same architecture trained as a regressor, and the margin grows exactly where the motion is least determined.

Part 1: The generative formulation

Same-architecture regressor, seeds, mean-of-K and best-of-K. See G2.

Model9-colOccluded
Same-architecture L2 regressor57.5252.51
TrackDiT, one sample (8 seeds)58.27 ± 0.0754.06 ± 0.12
TrackDiT, mean of 858.3054.28
TrackDiT, best of 8 (oracle)59.9755.49

TrackDiT is +0.81 [+0.40, +1.24] above the regressor overall and +1.57 [+1.12, +2.04] on occluded points.

"One random sample should not beat the average prediction."

  • The regressor is trained precisely to output the conditional mean, and it is 0.8 below a single TrackDiT sample.
  • Averaging 8 TrackDiT samples gives the same accuracy as one sample (58.30 vs 58.27). TrackDiT concentrates its samples where the evidence is strong and spreads them where it is weak: the sample spread is 1.40x [1.19, 1.66] larger on hidden than on visible points.
  • A single sample is therefore all that is needed in practice, and the best-of-8 oracle (60.0) shows further headroom from sample selection.

Ambiguity, not only cues from co-visible points. Both are at work, by design.

  • Co-visible points: spatial attention transfers motion from co-visible points on the same object (Section 4.5).
  • Where those cues run out: the generative objective pays off there. Against the regressor, TrackDiT's advantage grows from visible points (+0.63) to points occluded in frame (+1.02) and out of frame (+1.90). It also grows with the length of the hidden run, from +1.14 for 1-4 frames to +1.96 for 17-48 frames.
  • Motion magnitude: over hidden runs, TrackDiT samples reproduce the true path length (ratio 0.99 of GT). The regressor's paths are 17% too long (1.17; difference -0.175 [-0.235, -0.129], consistent across all 11 subsets).
  • Large errors: on out-of-frame points, TrackDiT has far fewer large errors (16.4% vs 22.6% beyond four times the largest APD threshold).
  • Uncertainty for free: a regressor cannot provide one, but the spread of TrackDiT's samples predicts its error (Spearman 0.52 [0.49, 0.54]; AUROC 0.77 [0.76, 0.78] for detecting the worst error quartile).

Novelty. We see two contributions.

  • The analysis. End-to-end 3D trackers are clearly weaker 2D trackers than a dedicated one (Table 1). The visible / occluded decomposition (Table 2) shows that a modern lift already wins on visible points and loses only behind occlusions.
  • A simple completion model that closes exactly that gap. The new experiments show its gain cannot be reproduced by any of:
    • simple fills: +2.5 over the best one;
    • existing 3D trackers on identical geometry: +2.2 to +5.7;
    • a same-architecture regressor: +0.8.

Part 2: Speed, scaling and dense tracking

Runtime and memory against 4RC, V-DPM and SpatialTrackerV2. See G4 (same H100, one 64-frame clip per benchmark).

  • Fastest end to end: including the 2D tracker and the geometry model, TrackDiT is the fastest on every benchmark. It takes 5.7-7.2 s per clip, against 9.1-11.2 s for 4RC, 14.8-18.0 s for SpatialTrackerV2 and 125-308 s for V-DPM.
  • Small correction model: our model alone takes 0.8-1.8 s and at most 2 GiB, which is 14-26% of the end-to-end time.
  • Peak memory is set by the dense 2D tracker. CoWTracker computes correspondences for every pixel in one pass, which is the source of the pipeline's 2D accuracy (Table 1) and makes dense tracking free per query. The stages run sequentially, and the correction model adds at most 2 GiB.
  • Fewer steps: accuracy remains similar with fewer Euler steps (reply to B58L, W6), so the correction model's cost can be reduced further if needed.

How many tracks are processed together, and the effect on accuracy. At test time, each call refines up to 1,024 query tracks together with a fixed grid of 512 support tracks. The model is trained with 128 tracks, and accuracy is the same for any call size:

Query tracks per call1282565121024
9-col58.2058.0758.2658.33
Occluded53.4153.4253.7154.08

Larger calls help occluded points (+0.67 [+0.25, +1.10] from 128 to 1,024), because more co-visible points are available.

Dense tracking.

  • The 2D tracker is already dense: CoWTracker tracks every pixel in one pass, so its time and memory do not depend on the number of queries (the same 28.4 GiB for 672 and 1,505 queries).
  • Chunking: in the correction model, spatial attention is bounded by processing queries in chunks that share the support grid, so cost grows linearly with the number of queries.
  • Already in the benchmark: this is how our evaluation handles the SynthVerse navigation clips, which have up to about 90,000 query points per clip and account for half of all refiner calls.
  • Long, real clips: a 300-frame TAPVid-3D clip with 1,243 tracks takes 6.5 s and 6.2 GiB in one call.

Part 3: Training data and simple fixes

SynthVerse. See G3.

  • Trained without SynthVerse, TrackDiT is zero-shot on SynthVerse, exactly like 4RC and V-DPM.
  • It still scores 57.3 overall and 47.2 on SynthVerse, above 4RC (54.5 / 45.1) and V-DPM (53.9 / 44.9).
  • Its accuracy on the real TAPVid-3D subsets is unchanged (88.2 vs 88.1).

Q3. Keep the tracker's ray and interpolate depth from nearby visible frames. See G1. This fix scores 53.27 (43.05 on occluded points). The best world-space fill scores 55.82; TrackDiT scores 58.33 (54.08 on occluded points).

Reviewer_xK7SOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper takes the position that 3D point tracking can be accomplished by chaining together (really good)

  • 2D tracking plus
  • occlusion estimation plus
  • depth estimation plus
  • pose estimation plus
  • their own proposed 3D track-inpainting (for points while they are occluded).

I like the idea of trajectory-completion for the 3D point tracking task, and I like that they're doing it as a refinement rather than a regression. The experimental protocol is less convincing, as I'll describe below. Overall, I'm prepared to lean in favor of the submission, but it is not clear cut that this paper will be cited much, since it's not the first to advocate such a multi-specialists approach (in contrast to single generalists).

Critical strengths

4D tracking is an important problem, and some systems that try to do it end-to-end are massive, or slow, or closed-weights (not replicable) or all of the above. Hybrid-models can be more like a pipeline.

The authors ask two reasonable questions, that are good common-sense: Is it so hard to interpolate a fragment of a 3D point trajectory (when training data exists for similar situations) and does appearance information help with that interpolation?

I like the visible-versus-occluded split.

Critical weaknesses

One easy improvement is to measure both regular and 3D Average Jaccard. The latter comes from TAPVid-3D. While that is cited here, it offers the upstream versions of Aria Digital Twin and PStudio, but before St4RTrack converted them into world-frame coordinates. It also has a DriveTrack split, which would help show the proposed work on real data. Finally, it also has 2D-trackes-then-lifted baselines, so the idea has certainly been around. It just obviously depends on when and for how long does the leading 2D tracker pull ahead.

Aren't SpatialTrackerV2 and TAPIP3D lift-style trackers with occlusion reasoning? But I don't see them in the comparisons?

TrackDiT teams up with VGGT-Ω, but teaming anything with VGGT-Ω is a good idea, so is it fair to credit TrackDiT for its effectiveness? TAPIP3D, SpatialTrackerV2 and DELTA could all use VGGT-Ω's outputs so they all get the same geometry foundation. That may lay bear how much is tracking vs how much is geometry.

While the authors argue that appearance is not needed for trajecotry refinment, they don't test whether it would help. It wouldn't be hard, and could be a factor for refining the data-association of points before and after an occlusion. TrackDiT beats 4RC and the other end-to-end models on occluded points, and those models do use appearance. But they also differ in training data and geometry.

TrackDiT trains on SynthVerse, and 7 out of 9 columns in the headline average are SynthVerse subsets. The authors are less generous when explaining that 4RC’s win on the interaction subset is based on its training data.

Questions for authors

The proposed trained "denoiser" is built for Cowtracker. Would it work with other 2D trackers without special retraining? E.g. CoTracker3? Is refinement confined to this pipeline, or could it learn to improve SpatialTrackerV2 (so presumably needing smaller adjustments)?

The occluded 3D points are first reset to the per-track visible-anchor median before being passed to the denoiser. That is certainly allowed, but the Simple Baseline doesn't get that opportunity, and instead keeps the occluder’s depths which really are big shocks in position. Besides, linear and spline interpolation in world space would be equally sensible baselines.

How do the scores look without global scaling against ground-truth, but say, using a median-scaling protocol?

Generative models can be inconsistent. Are we seeing the results of one seed?

2 comments
AuthorsOpenReview ↗

Response to Reviewer xK7S

We thank the reviewer for the positive view of trajectory completion as refinement, and for the concrete suggestions. Every request has been run; shared results are in the General Response (G1-G4).

Part 1

W1. TAPVid-3D, 3D-AJ, upstream ADT / PStudio and DriveTrack. We evaluated on TAPVid-3D minival with the official protocol: camera frame, median scaling, official scorer, 50 clips per source, full-length clips. We used both GT and predicted (VGGT-Ω) intrinsics.

IntrinsicsMethod3D-AJ3D-APDOA3D-APD ADT / DriveTrack / PStudio
GTSimple Baseline (lift)24.4734.3388.845.38 / 27.83 / 29.79
GTTrackDiT25.2835.3788.846.33 / 28.41 / 31.38
VGGT-ΩSimple Baseline (lift)16.1822.2788.844.23 / 14.41 / 8.18
VGGT-ΩTrackDiT16.7122.9188.845.16 / 14.97 / 8.58
  • GT intrinsics: 3D-AJ +0.81 [+0.31, +1.32], 3D-APD +1.04 [+0.29, +1.76].
  • VGGT-Ω intrinsics: 3D-AJ +0.53 [+0.20, +0.86], 3D-APD +0.63 [+0.14, +1.12].
  • Per source: TrackDiT improves 3D-APD on all three sources in both settings, including DriveTrack.

TAPVid-3D scores positions only on GT-visible points. Completing occluded points, TrackDiT's main contribution, is therefore measured by the world-frame protocol of Table 3. That protocol uses the same TAPVid-3D ADT and PStudio sequences converted to world coordinates by St4RTrack.

On prior lifting baselines:

  • We will cite the 2D-tracks-then-lift baselines of TAPVid-3D; our Simple Baseline belongs to this family.
  • Our finding is that, with today's 2D trackers and reconstructors, this family already matches end-to-end models on visible points (Table 2).
  • A light completion model then turns it into the state of the art overall.
  • That TrackDiT benefits from the leading 2D tracker is a strength, not a dependence:
    • The framework is modular, so every advance in 2D tracking or reconstruction carries over without retraining.
    • Q1 shows this with CoTracker3 and SpatialTrackerV2 tracks.
    • Its gains already hold today on identical geometry (W3).

W2. SpatialTrackerV2 and TAPIP3D. Both are in Table 3 (rows v and vi) and in the qualitative comparison of Figure 4. We have now also evaluated them on exactly our geometry (W3) and per point class (reply to B58L).

W3. Same VGGT-Ω geometry for TAPIP3D, SpatialTrackerV2 and DELTA. We gave all three the dense VGGT-Ω depth and cameras that TrackDiT uses, on all 690 sequences:

MethodGeometry9-colVisOccMedian-scaledTrackDiT ahead by
TrackDiTVGGT-Ω58.3362.1954.0846.03-
TAPIP3DVGGT-Ω56.1460.0951.3544.12+2.20 [+1.61, +2.86]
SpatialTrackerV2VGGT-Ω54.0058.4248.9042.90+4.33 [+3.62, +5.13]
DELTAVGGT-Ω52.6158.8945.5740.19+5.73 [+4.96, +6.47]
Simple Baseline + per-track medianVGGT-Ω55.8260.3449.1443.89+2.51 [+1.94, +3.09]
  • Geometry helps the baselines, but TrackDiT stays first. VGGT-Ω lifts TAPIP3D from 50.0 to 56.1 and SpatialTrackerV2 from 51.2 to 54.0. TrackDiT still leads every method, with the largest margins on occluded points (+2.7 to +8.5).
  • Tracking adds little on the same geometry; TrackDiT adds +2.5. Existing 3D tracking adds at most +0.3 over the plain lift, whereas TrackDiT adds +2.5. TrackDiT's gain is therefore not a geometry effect.
  • With DA3 geometry as well. TrackDiT with DA3 (57.0, Table 3) also outperforms all baselines.

Part 2

W4. Appearance. TrackDiT reaches the state of the art from geometry alone. This keeps it small and lets it work with any tracker and reconstructor. Appearance is complementary. We trained the main recipe with frozen DINOv3 or ConvNeXtV2 features sampled at each track position, simply concatenated to the geometric tokens:

Model9-colOccluded (11 subsets)TAPVid-3D avg (4 real subsets)
TrackDiT (geometry only)58.0354.7588.09
+ DINOv3 features57.6354.8088.32
+ ConvNeXtV2 features57.5554.6588.26
  • Even simple concatenation keeps the model at the same level: the overall change is within 0.5 points.
  • On real data it goes slightly up: the TAPVid-3D average rises with both feature types (higher on ADT, PStudio and PointOdyssey).
  • Future work: a dedicated design, such as using appearance to re-associate a point after an occlusion as the reviewer suggests, is a promising extension. We will add this discussion.

W5. SynthVerse in training and in 7 of 9 columns. See G3. Without SynthVerse, the SynthVerse columns are zero-shot for TrackDiT as for 4RC. TrackDiT still leads 4RC overall (57.3 vs 54.5) and on the SynthVerse average (47.2 vs 45.1), and its real-data accuracy is unchanged.

Q1. Other 2D trackers without retraining; refining SpatialTrackerV2. The same model is used throughout, never retrained:

Input tracksInput (9-col)+ TrackDiTGain, allGain, occluded
CoWTracker 2D + VGGT-Ω lift55.8258.33+2.51 [+1.94, +3.09]+4.95 [+4.22, +5.67]
CoTracker3 2D + VGGT-Ω lift (zero-shot)55.1257.01+1.90 [+1.33, +2.47]+3.09 [+2.25, +3.97]
SpatialTrackerV2 2D + VGGT-Ω lift (zero-shot)51.6654.87+3.21 [+2.69, +3.72]+4.99 [+4.27, +5.71]
SpatialTrackerV2 3D tracks (zero-shot)53.9354.14+0.21+0.57 [+0.24, +0.91]
  • CoTracker3: TrackDiT transfers without retraining. Swapping in this tracker it was never trained on, TrackDiT still turns its lift (55.1) into 57.0, ahead of every published 3D tracker in Table 3.
  • SpatialTrackerV2's 2D tracks: lifted and completed by TrackDiT, they beat SpatialTrackerV2's own 3D output on the same geometry, 54.87 vs 53.93.
    • Differences: +0.94 [+0.55, +1.34] overall, +1.51 [+1.04, +2.01] on occluded points and +1.09 [+0.77, +1.44] median-scaled.
    • The same correspondences, completed by TrackDiT, are better than SpatialTrackerV2's own 3D tracking.
  • SpatialTrackerV2's 3D tracks: TrackDiT also refines them directly, without fine-tuning, even though SpatialTrackerV2 has its own occlusion reasoning. Occluded points improve by +0.57 [+0.24, +0.91] and median-scaled accuracy by +0.21 [+0.11, +0.31].

Q2. The Simple Baseline keeps the occluder depth; linear and spline baselines.

  • The Simple Baseline is deliberately the naive lift (Section 3, Figure 1). It measures what off-the-shelf components give without any reasoning about occlusion, which is the basis of Observations I and II.
  • The per-track median reset is part of TrackDiT's input preparation (Appendix A).
  • To answer the question directly, G1 evaluates the lift with the median reset itself, linear interpolation and a cubic spline in world space, and ray-preserving depth interpolation.
  • The best fill reaches 55.8; TrackDiT is +2.5 above it, and +5.0 on occluded points.

Q3. Median scaling instead of global scaling. Every new table includes the median-scaled metric. The ranking is unchanged:

MethodTrackDiTTAPIP3DLift + medianSpatialTrackerV2DELTATraceAnything
Median-scaled 9-col46.0344.1243.8942.9040.1934.36

TAPIP3D, SpatialTrackerV2 and DELTA use VGGT-Ω geometry here. The TAPVid-3D results in W1 also use the official median scaling.

Q4. One seed? Over 8 sampling seeds, TrackDiT scores 58.27 ± 0.07 (range 58.19-58.38). Results are stable, and one sample matches the 8-sample ensemble (58.30); see G2.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.