- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The following is a brief summary of the TrackDiT paper as understood by this reviewer:
- The authors show that 3D trackers are not good at transferring their tracks back to the 2D image, despite claims of good 3D tracking and camera pose estimation.
- They show a simple baseline of 2D tracking with 3D lifting out-competes these 3D trackers on visible points but struggles when points undergo occlusions due to other objects and out-of-frame.
- The authors introduce a new modular method which utilises the best of 2D trackers and 3D trackers to lift 2D tracks to 3D before using a flow matching design to de-noise tracks in 3D so that they perform better across occlusion gaps.
- They show consistent improvement in 3D tracking over prior state-of-the-art models.
Critical strengths
The following is a list of strengths for the paper:
- The paper is well presented and easy to understand. The authors start with a thorough motivation for their work by going through the pros and cons of 2D and 3D trackers, with a comparison to a simple baseline. The method then follows on nicely from this start before results are presented that back up their claims well.
- The authors include interesting and informative diagnostics of point tracking performance split by visible and occluded points. This is nice to see as it points out that they are very different challenges and further supports the gap that their work looks to fill.
- It is good to see the comparison of 2D and 3D tracking. Some works (e.g. SpatialTracker) have made the claim that tracking is easier to conduct in the 3D space, but not many have thought about how this translates back to the 2D plane once we have good 3D tracks and whether these models are even very good at this new task.
- The method section is well motivated, supported by simple and effective figures as well as being a fairly elegant solution to the problem.
- The authors show state-of-the-art performance across a number of benchmarks and against a large number of 3D trackers.
Critical weaknesses
The following is a list of things that this reviewer considers weaknesses of this paper:
- It would be more complete to see a simpler baseline of simple interpolation to fill the occluded regions in 3D. It is likely that this will be much worse but it will help the reader understand the kind of motion that is found in the benchmarks and whether the diffusion process is learning to predict any complex motion.
- A more detailed discussion on the process of handling out-of-frame points would be good to see. Do 4RC, Any4D and TraceAnything already handle this? Or are the authors assuming something for these models when points leave the frame?
- Although the main table shows good improvements against 3D trackers, it would also be good to see a selection of 2D trackers in this evaluation. This is not a strong weakness, but more of an interesting point to make the paper more complete.
- In the implementation details, the authors mention training on 48-frame windows. However, there does not seem to be any analysis of point tracking accuracy over regions shorted and longer than this training window. This seems like an important ablation to include. The fact that large residuals are left out of training potentially points to the method maybe struggling quite a lot in long-term occlusions, especially out of frame.
- The weighting of the loss function is understandable but the paper is missing a proper ablation on how these weightings were chosen. For example, why are occluded and out-of-frame weighted the same? As surely out-of-frame is harder than simply occluded within the frame?
- Although the ablations are good, they do appear to be quite limited. For example, there are no ablation on the number of layers, dimension sizes, number of Euler steps, region weightings, training window size etc.
- The qualitative examples look really nice. However, there does not seem to be many with tracks leaving the frame or being occluded through complex motion. If these were included, then they would be more convincing.
Questions for authors
- How are out-of-frame points handled by the baselines?
- Is simple interpolation between visible points evaluated as a baseline anywhere?
- Why are occluded and out-of-frame weighted the same in the loss?
- How does tracking accuracy fair when the model needs to predict large residuals for refinement in the flow matching stage? E.g. if the actual motion is very complex and far from the noise/cold start then does it struggle?
2 comments
Q4. Large residuals. Tokens grouped by the lift's error before refinement:
| Lift error | 2nd quartile | 3rd quartile | Largest quartile | Largest quartile, hidden points |
|---|---|---|---|---|
| TrackDiT gain | +2.20 | +3.83 | +9.59 [+8.13, +11.04] | +12.35 [+9.70, +14.10] |
The model does not struggle when large corrections are needed; its gain grows with the size of the required correction, and is largest (+12.4) for hidden points with the largest lift errors, which is exactly the case the reviewer asked about.
Response to Reviewer B58L
We thank the reviewer for the positive assessment and the many concrete suggestions. We ran all requested analyses and ablations.
Part 1
W1 / Q2. Simple interpolation baseline. See G1.
W2 / Q1. How the baselines handle out-of-frame points. No assumption is made for any baseline; each outputs a 3D point for every query in every frame:
Accuracy per point class, with each method under its own alignment:
TrackDiT is best in every class, and by the widest margin out of frame: +8.4 [+6.3, +10.9] over the lift and +6.8 [+4.7, +9.1] over SpatialTrackerV2.
W3. 2D trackers in the evaluation. We added three rows, each a 2D tracker followed by our lift:
Part 2
W4. Shorter vs. longer than the 48-frame training window; long occlusions. TrackDiT's gain over its input, grouped by how long a point stays hidden:
The gain stays the same for occlusions longer than the training window. We also retrained with 24- and 64-frame windows (ablation recipe, all 11 subsets). Both stay within 0.4 points of the 48-frame model (24 frames: +0.32 [+0.04, +0.62]; 64 frames: +0.18 [-0.15, +0.49]), so the method is insensitive to the window length.
W5 / Q3. Region weights; occluded vs. out-of-frame. Ablation recipe, all 11 subsets:
W6. More ablations. Ablation recipe (all 11 subsets), and the number of Euler steps on the main model (9-col):
W7. Qualitative examples. We refer the reviewer to the supplementary videos, which cover both kinds of occlusion under complex motion:
swing,bmx-trees(the cyclist passing behind the tree trunks) ande-bike.parkour,tennis-vestandrollerblade, where limbs repeatedly occlude each other and the person is occluded by objects.In all of them, the traces stay smooth through the occlusions, without the flicker or flying points of the raw lift, which shows that the model reasons correctly about the hidden motion. In the revision we will add more scenes and viewpoints, focusing on points that leave the frame and on long occlusions.