acceptodds
Under review as a conference paper at ICLR 2027

Reading Point Tracks from Frozen Video Diffusion Transformers

Abstract

We show that readout design substantially improves point tracking from frozen video diffusion transformers, without training a tracking network. Our readout combines confidence-weighted attention heads, bilinear source sampling, local position decoding and visibility estimation from one transformer block. All settings are selected on 20 labelled synthetic clips and transferred unchanged to real videos. CogVideoX-2B reaches 5.21 average Jaccard (AJ) on TAP-Vid DAVIS and 44.00 on Kinetics. Evaluating released HeFT on the same videos, queries and checkpoints with the official TAP-Vid first-query metric establishes gains on both benchmarks for CogVideoX and Wan, including +18.62 and +12.77 AJ on Kinetics. Controlled ablations on both backbones show why localisation improves: bilinear source sampling and local decoding reinforce each other, and most of the local-decoding gain comes from predicting positions between token centres. On CogVideoX, confidence weighting further improves head aggregation, whereas tested block combinations give no clear gain on the synthetic selection set. Across six checkpoints from three generator families, larger models do not consistently improve tracking under this readout. Our readout improves zero-shot correspondence estimation while keeping the video generator frozen.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
point tracking, emergent correspondence, video diffusion models, zero-shot, training-free, attention readout, test-time adaptation, TAP-Vid

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_uwvHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a method for point tracking by using features from a frozen video diffusion transformer. Given an anchor and a target frame, the key/query matrices are read out after particular attention blocks, a confidence weighted aggregation is performed before a nearest neighbor matching. Visibility estimation is performed based on peak scores and cycle consistency. The different hyperparameters (which block, which diffusion step, confidence and visibilit thresholds) are optimized based a small synthetic dataset and used for real evaluations. The method is demonstrated on top of CogVideoX, Wan, and Cosmos.

The paper is a thorough empirical study, but its conceptual novelty is incremental. It applies known readout operations (bilinear sampling, local weighted mean, head weighting, cycle consistency) to a new class of frozen features (video DiT attention) and carefully ablates their interactions.

Critical strengths
  • The paper presents a training free approach where only a hyperparameter selection based on a small synthetic dataset is needed.
  • The evaluations and ablations are very extensive and detailed.
  • Some experimental results are interesting, e.g., CogVideoX-5B actually performs worse than CogVideoX-2B, Wan-14B is statistically indistinguishable from Wan-1.3B, and Cosmos-14B's gain is marginal. Hence there is no direct corralation between model size and the tracking accuracy.
Critical weaknesses
  1. The idea of reusing vision transformer or diffusion features is well studied and hence individual components are well known.

The paper's novelty therefore lies in (a) assembling these into a coherent pipeline, (b) the synthetic-selection-then-transfer protocol, and (c) the controlled attribution of which pieces matter. This is more of an empricial study/contribution rather than a new technical contribution.

  1. There is a mismatch between the HeFT comparisons presented in the paper vs the original paper which the authors also mention.

  2. Similar to other work which uses pre-trained features, the gap to trained trackers is still large. It is competitive among zero-shot methods, but a dedicated tracker is substantially better.

  3. Visibility estimation remains challenging as adding oracle visible estimations result in significant gains.

  4. While, the main text is 9 pages, the appendices run to 35 more pages. Not sure how to evaluate the length of the paper vs the technical contribution.

  5. There are no visual video tracking results provided.

Questions for authors
  1. A single diffusion timestep with clean latents is used. Can the authors comment on whether multi-diffusion steps might help or whether they observe a trend with respect to which diffusion step to use?

  2. The experiment results suggest that larger model sizes do not directly contribute to better performance. Do the authors have any insights why this may be the case?

  3. The paper tests some video diffusion models but not the most recent image models. It's not clear yet whether a video model is better suited for this task as opposed to an image model.

Reviewer_C4tdOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how to extract accurate point trajectories from frozen video diffusion transformers without training a dedicated tracking network. Building upon existing correspondence methods such as DiffTrack and HeFT, the authors propose an improved readout combining confidence-weighted attention-head aggregation, bilinear source sampling, local sub-token position decoding, and visibility estimation through matching confidence and cycle consistency. The method selects its hyperparameters using only 20 labeled synthetic videos and transfers them to real-world tracking benchmarks without further tuning.

Critical strengths
  1. Strong empirical improvements over closely related methods. The paper compares against released HeFT implementations under matched videos, queries, and evaluation metrics;
Critical weaknesses
  1. HeFT [1] already uses soft-argmax for sub-token localization and forward–backward consistency, so these operations themselves are not novel;
  2. missing cited previous work [2];

[1] Denoise to Track: Harnessing Video Diffusion Priors for Robust Correspondence [2] Video Diffusion Models Excel at Tracking Similar-Looking Objects Without Supervision

Questions for authors

A large fraction of the localization improvement appears to arise from correcting token-center coordinates and recovering sub-token positions. Can the authors quantify how much of the improvement over HeFT comes from coordinate handling, localization, head weighting, and visibility estimation under a fully shared feature extraction and evaluation protocol? Would applying the proposed local decoder directly to HeFT's selected features achieve comparable performance?

Reviewer_uXgSOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper reads point tracks from the query/key features of frozen video diffusion Transformers. It predicts positions and visibility by weighting different heads according to confidence, sampling at continuous positions, and using local weighted decoding. The authors select hyperparameters on 20 labeled synthetic videos and then directly apply them to real videos, without training a tracking network throughout the process. CogVideoX-2B obtains 45.21 and 44.00 AJ on DAVIS and the recovered 1,144 Kinetics videos, respectively. The paper also analyzes the sources of improvement through comparisons with HeFT and component ablations (p.3–9; Appendices B–H).

Critical strengths

Under the official DAVIS first protocol, CogVideoX and Wan improve over the released HeFT implementation by about 13 and 9 AJ points, respectively. The advantage remains after unifying inputs and physical query positions. The Kinetics comparison covers all 1,144 recovered videos. This makes the results more convincing than directly comparing the reported scores in different papers (p.6, Table 2; p.19–20, Tables A6–A8).

On CogVideoX, local-mean decoding brings a clear improvement, while rounding the predictions back to token centers loses most of the gain. Together with the repeated experiments on Wan, this shows that predicting positions between tokens is indeed a main source of the gain. The interaction between source sampling and local decoding also explains why replacing only one operation may not show the same improvement (p.7, Table 3; p.25–26, Tables A14–A17).

Critical weaknesses

Query/key matching is inherited from earlier work, and local weighted decoding is also an established operation in tracking. The confidence-based head weighting is useful, but the method as a whole feels incremental. For me, the value is mainly in how these pieces are combined and analyzed.

The original query descriptor stays fixed throughout tracking, so appearance changes are still difficult to handle. Peak scores and cycle consistency can also accept wrong matches. Appendix I reports lower accuracy for large motion and reappearance after occlusion. Robustness in these cases is still a limitation of the current readout

Questions for authors

When the anchor frame is put first in each interleaved chunk, do the temporal position encodings use the original video frame indices or the positions within the chunk? I would like to understand how the model represents the time gaps between the sampled frames

For Cosmos, key/key matching lowers accuracy in your readout (Table A27), while replacing key/key with query/key lowers accuracy in HeFT. How do you interpret these opposite effects?

What is the main reason for keeping the query descriptor fixed throughout tracking, given the difficulty with appearance changes discussed in Section 6?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.