- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a framework for generating pseudo-labels for point tracking from an ensemble of pretrained trackers. The Candidate Curator evaluates candidate predictions through local visual evidence, inter-candidate interactions, and temporal attention. The Coordinate Recomposer subsequently refines the selected candidate using query-conditioned local correspondence and temporal context, allowing the output coordinate to depart from the discrete candidate set. Visibility is obtained through rank-weighted aggregation of teacher predictions. The generator is trained on synthetically perturbed ground-truth trajectories from Kubric and evaluated using predictions from six pretrained trackers. Experiments on ITTO-MOSE, TAP-Vid Kinetics, and TAP-Vid DAVIS show modest improvements over verifier-guided pseudo-labeling. The generated labels are also used to fine-tune five trackers on RGBD1K, yielding improvements in most tracker–benchmark combinations.
Critical strengths
-
The method addresses a concrete limitation of discrete candidate selection. Selecting a reliable candidate and then refining its coordinate is well motivated: even a correctly selected candidate can retain localization error. The proposed use of local correspondence provides visual evidence for this correction. Appendix A.1 offers a useful decomposition under the same candidate set: DAVIS localization accuracy increases from 81.2 for the Verifier to 81.4 for the Curator and then to 81.8 with the Recomposer. This supports the contribution of both stages.
-
The evaluation covers both pseudo-label quality and downstream utility across multiple tracking architectures. The paper evaluates three real-world benchmarks and fine-tunes five student trackers. Most combinations improve; for example, BootsTAPIR’s AJ on ITTO-MOSE increases from 36.9 to 38.8, and CoTracker3’s AJ on DAVIS increases from 63.8 to 64.8. Appendix F additionally investigates candidate interactions, local evidence, temporal attention, and patch-to-patch correspondence. These experiments provide useful evidence that the framework can benefit different tracking architectures.
Critical weaknesses
-
The independent benefit and robustness of the proposed generator are not sufficiently established. Relative to the Verifier, Table 1 reports AJ improvements of 0.2, 0.3, and 0.6 percentage points on ITTO-MOSE, Kinetics, and DAVIS, respectively. The corresponding localization improvements are 0.2, 0.0, and 0.6 points. These results are encouraging, but no variation across training seeds or paired video-level confidence intervals is reported, making their robustness difficult to assess. More importantly, the downstream experiments change several factors simultaneously. The proposed pipeline includes additional real-world training, quality filtering, synthetic–real mixing, and a query curriculum. Table 2 therefore demonstrates the utility of the overall adaptation procedure, but does not isolate the contribution of the proposed pseudo-label generator. Table 4 ablates adaptation components, yet does not compare alternative pseudo-label sources under an otherwise identical training protocol. Improvements are also not universal: Track-On-R’s ITTO-MOSE AJ decreases from 45.5 to 45.3, and its localization accuracy decreases from 61.4 to 60.8. A controlled comparison that changes only the pseudo-label source is essential to substantiate the claimed downstream advantage over verifier-guided supervision.
-
The temporal analysis does not adequately distinguish correction of switching artifacts from genuine motion or generic smoothing. Appendix A.2 defines the switching jump as the 95th percentile of predicted frame-to-frame displacement at candidate-switching transitions. This quantity includes genuine point motion; reducing it does not necessarily imply reducing tracking error and could also reflect attenuation of actual motion. Furthermore, the Verifier and Curator switch at different transitions, so their conditional displacement statistics are computed over different event sets. The Curator–Recomposer comparison uses the same switching events and is therefore more interpretable, but still does not account for ground-truth motion. Figure 7 provides complementary ground-truth-relative error curves, but does not replace an aggregate evaluation. A more direct measurement would assess the error in predicted displacement over a common set of transitions. The paper also lacks comparisons against the Verifier followed by the same Recomposer, simple temporal smoothing, or a standard local refinement module. Consequently, the current evidence does not fully establish whether the proposed joint design specifically addresses switching-induced errors beyond what these alternatives could achieve.
Questions for authors
-
Can you compare Random, Verifier, Curator-only, and the complete generator while holding student initialization, training data, query sampling, filtering, and optimization budget fixed? A controlled experiment on at least one representative student would materially affect my assessment.
-
Do the temporal improvements persist when evaluated using ground-truth-relative displacement error over the same transitions? How does the proposed model compare with the Verifier followed by the same Recomposer, and with a simple smoothing or local-refinement baseline?
-
Was the Verifier trained with the same candidate augmentation distribution, including the additional near-confusing trajectories? Were visibility aggregation rules matched between methods? Please also provide uncertainty estimates for the principal improvements, through repeated training runs and/or paired video-level confidence intervals.
-
Was the visibility temperature selected independently of the reported evaluation benchmarks? Appendix C motivates this choice using DAVIS and ITTO-MOSE results, so the validation protocol should be clarified. Please also explain whether Table 7(a) includes the Recomposer and why the full-model AJ differs between Tables 1 and 7.