- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes a method for point tracking by using features from a frozen video diffusion transformer. Given an anchor and a target frame, the key/query matrices are read out after particular attention blocks, a confidence weighted aggregation is performed before a nearest neighbor matching. Visibility estimation is performed based on peak scores and cycle consistency. The different hyperparameters (which block, which diffusion step, confidence and visibilit thresholds) are optimized based a small synthetic dataset and used for real evaluations. The method is demonstrated on top of CogVideoX, Wan, and Cosmos.
The paper is a thorough empirical study, but its conceptual novelty is incremental. It applies known readout operations (bilinear sampling, local weighted mean, head weighting, cycle consistency) to a new class of frozen features (video DiT attention) and carefully ablates their interactions.
Critical strengths
- The paper presents a training free approach where only a hyperparameter selection based on a small synthetic dataset is needed.
- The evaluations and ablations are very extensive and detailed.
- Some experimental results are interesting, e.g., CogVideoX-5B actually performs worse than CogVideoX-2B, Wan-14B is statistically indistinguishable from Wan-1.3B, and Cosmos-14B's gain is marginal. Hence there is no direct corralation between model size and the tracking accuracy.
Critical weaknesses
- The idea of reusing vision transformer or diffusion features is well studied and hence individual components are well known.
The paper's novelty therefore lies in (a) assembling these into a coherent pipeline, (b) the synthetic-selection-then-transfer protocol, and (c) the controlled attribution of which pieces matter. This is more of an empricial study/contribution rather than a new technical contribution.
-
There is a mismatch between the HeFT comparisons presented in the paper vs the original paper which the authors also mention.
-
Similar to other work which uses pre-trained features, the gap to trained trackers is still large. It is competitive among zero-shot methods, but a dedicated tracker is substantially better.
-
Visibility estimation remains challenging as adding oracle visible estimations result in significant gains.
-
While, the main text is 9 pages, the appendices run to 35 more pages. Not sure how to evaluate the length of the paper vs the technical contribution.
-
There are no visual video tracking results provided.
Questions for authors
-
A single diffusion timestep with clean latents is used. Can the authors comment on whether multi-diffusion steps might help or whether they observe a trend with respect to which diffusion step to use?
-
The experiment results suggest that larger model sizes do not directly contribute to better performance. Do the authors have any insights why this may be the case?
-
The paper tests some video diffusion models but not the most recent image models. It's not clear yet whether a video model is better suited for this task as opposed to an image model.