acceptodds
Withdrawn from ICLR 2027

Generalized Pseudo-Label in Point Tracking

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Reliable trajectory supervision is fundamental to robust point tracking, yet dense point-level annotations for real-world videos remain severely limited. To alleviate this scarcity, recent studies have increasingly explored two pseudo-labeling paradigms: dedicated methods that derive pseudo-labels from unlabeled videos and self-training approaches that reuse a tracker’s own predictions as pseudo-labels. These approaches rely on different assumptions and mechanisms, resulting in distinct prediction characteristics and varying reliability across video sequences, frames and points. To account for this variability, we propose a pseudo-label generation framework consisting of a Candidate Curator and a Coordinate Recomposer. Specifically, the Candidate Curator constructs context-aware representations from local evidence around the query and candidate points. It then reasons over temporal context and frame-level candidate relations to select reliable candidates for each point and frame. However, independent frame-wise selection can introduce temporal discontinuities in the pseudo-label trajectory. We therefore introduce the Coordinate Recomposer, which models query-conditioned spatiotemporal correspondence to recompose the selected candidate into a temporally coherent point prediction. Extensive experiments demonstrate that the proposed framework produces more accurate pseudo-labels and provides effective supervision for real-world point-tracking adaptation. Fine-tuning existing point trackers with our generated pseudo-labels further improves their performance on real-world videos.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Point Tracking, Pseudo-Labeling

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_U93FOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a framework for generating pseudo-labels for point tracking from an ensemble of pretrained trackers. The Candidate Curator evaluates candidate predictions through local visual evidence, inter-candidate interactions, and temporal attention. The Coordinate Recomposer subsequently refines the selected candidate using query-conditioned local correspondence and temporal context, allowing the output coordinate to depart from the discrete candidate set. Visibility is obtained through rank-weighted aggregation of teacher predictions. The generator is trained on synthetically perturbed ground-truth trajectories from Kubric and evaluated using predictions from six pretrained trackers. Experiments on ITTO-MOSE, TAP-Vid Kinetics, and TAP-Vid DAVIS show modest improvements over verifier-guided pseudo-labeling. The generated labels are also used to fine-tune five trackers on RGBD1K, yielding improvements in most tracker–benchmark combinations.

Critical strengths
  1. The method addresses a concrete limitation of discrete candidate selection. Selecting a reliable candidate and then refining its coordinate is well motivated: even a correctly selected candidate can retain localization error. The proposed use of local correspondence provides visual evidence for this correction. Appendix A.1 offers a useful decomposition under the same candidate set: DAVIS localization accuracy increases from 81.2 for the Verifier to 81.4 for the Curator and then to 81.8 with the Recomposer. This supports the contribution of both stages.

  2. The evaluation covers both pseudo-label quality and downstream utility across multiple tracking architectures. The paper evaluates three real-world benchmarks and fine-tunes five student trackers. Most combinations improve; for example, BootsTAPIR’s AJ on ITTO-MOSE increases from 36.9 to 38.8, and CoTracker3’s AJ on DAVIS increases from 63.8 to 64.8. Appendix F additionally investigates candidate interactions, local evidence, temporal attention, and patch-to-patch correspondence. These experiments provide useful evidence that the framework can benefit different tracking architectures.

Critical weaknesses
  1. The independent benefit and robustness of the proposed generator are not sufficiently established. Relative to the Verifier, Table 1 reports AJ improvements of 0.2, 0.3, and 0.6 percentage points on ITTO-MOSE, Kinetics, and DAVIS, respectively. The corresponding localization improvements are 0.2, 0.0, and 0.6 points. These results are encouraging, but no variation across training seeds or paired video-level confidence intervals is reported, making their robustness difficult to assess. More importantly, the downstream experiments change several factors simultaneously. The proposed pipeline includes additional real-world training, quality filtering, synthetic–real mixing, and a query curriculum. Table 2 therefore demonstrates the utility of the overall adaptation procedure, but does not isolate the contribution of the proposed pseudo-label generator. Table 4 ablates adaptation components, yet does not compare alternative pseudo-label sources under an otherwise identical training protocol. Improvements are also not universal: Track-On-R’s ITTO-MOSE AJ decreases from 45.5 to 45.3, and its localization accuracy decreases from 61.4 to 60.8. A controlled comparison that changes only the pseudo-label source is essential to substantiate the claimed downstream advantage over verifier-guided supervision.

  2. The temporal analysis does not adequately distinguish correction of switching artifacts from genuine motion or generic smoothing. Appendix A.2 defines the switching jump as the 95th percentile of predicted frame-to-frame displacement at candidate-switching transitions. This quantity includes genuine point motion; reducing it does not necessarily imply reducing tracking error and could also reflect attenuation of actual motion. Furthermore, the Verifier and Curator switch at different transitions, so their conditional displacement statistics are computed over different event sets. The Curator–Recomposer comparison uses the same switching events and is therefore more interpretable, but still does not account for ground-truth motion. Figure 7 provides complementary ground-truth-relative error curves, but does not replace an aggregate evaluation. A more direct measurement would assess the error in predicted displacement over a common set of transitions. The paper also lacks comparisons against the Verifier followed by the same Recomposer, simple temporal smoothing, or a standard local refinement module. Consequently, the current evidence does not fully establish whether the proposed joint design specifically addresses switching-induced errors beyond what these alternatives could achieve.

Questions for authors
  1. Can you compare Random, Verifier, Curator-only, and the complete generator while holding student initialization, training data, query sampling, filtering, and optimization budget fixed? A controlled experiment on at least one representative student would materially affect my assessment.

  2. Do the temporal improvements persist when evaluated using ground-truth-relative displacement error over the same transitions? How does the proposed model compare with the Verifier followed by the same Recomposer, and with a simple smoothing or local-refinement baseline?

  3. Was the Verifier trained with the same candidate augmentation distribution, including the additional near-confusing trajectories? Were visibility aggregation rules matched between methods? Please also provide uncertainty estimates for the principal improvements, through repeated training runs and/or paired video-level confidence intervals.

  4. Was the visibility temperature τ=5\tau=5 selected independently of the reported evaluation benchmarks? Appendix C motivates this choice using DAVIS and ITTO-MOSE results, so the validation protocol should be clarified. Please also explain whether Table 7(a) includes the Recomposer and why the full-model AJ differs between Tables 1 and 7.

Reviewer_FT8VOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper presents a pseudo-label generator for point tracking. A set of pre-trained trackers provide candidate tracks which are then curated and subsequently refined by the proposed method into a single point per frame. The method is trained with synthetic perturbation on Kubric, and then, for generalization, RGBD1K is pseudo-labeled and the off-the-shelf trackers are fine-tuned on it, so that the generator can learn from data closer to the eval setting (in terms of tracks and visuals). The experiments show that the pseudo-labels are slightly better than prior verifier-style work on three datasets.

Critical strengths
  • The method's motivation is clear: prior per-frame teacher selection can jump between frames and overall cannot be more accurate than the best teacher.

  • Moving from choosing teachers to some continuous version that can interpolate/refine is a logical next step of this line of work.

  • Table 2 shows gains across 5 current trackers when fine-tuned with the proposed label generator.

Critical weaknesses

The gains over the most related prior method, "Verifyer" (Aydemir et al.), are very small in Table 1. This raises two questions:

  • Do these differences hold up under multiple seeds? What is the variance of these results?
  • How would the results in Table 2 look if fine-tuned on Verifyer's pseudo labels? Do the gains in pseudo labels affect the final tracking performance?

Table 1 compares random (per frame (?), not clear) teacher selection, to Verifyer and the proposed method. There are other baselines that are very important to check:

  • Comparing Table 2 to Table 1, I am missing the comparison of the best tracker to the pseudo-labels. In a sense, this answers the question of what if we only label the data with a single tracker (the best one, ie Track-On-R). It looks like that in several metrics and datasets this actually performs better than the proposed method, so it is not clear if the proposed approach is actually more useful than labeling everything with just this tracker.
  • CoTracker3 uses per-batch teacher selection instead of per-frame. Does this make a difference? It should for sure have a much lower switch-rate (Fig 1) than 73%

The central claim in the paper is that discrete selection is the main problem in prior work. This needs to be validated better to confirm this hypothesis.

  • The curator's scores could be used to compute a weighted average rather than choosing the argmax.
  • One could compute the median of the candidates.
  • Overall, with gradient descent, and depending on the loss, the student in the random teacher setting (and eg CoTracker3) should learn the conditional mean (L2) or the conditional median (L1) of the teacher distribution. One could compute these as baselines too.

Compared to prior work, three changes are made: architecture, loss, and data are different. Where do the gains come from?

Questions for authors

There are several important questions that follow from the weakness section above:

  • Do the reported gains over "Verifyer" (Table 1) hold up when the experiment is run with multiple random seeds?
  • Does per-batch teacher selection inherently make a difference in performance compared to the per-frame selection used here?
  • How do the results look in Table 2 if the data is fine-tuned using Verifyer's pseudo-labels?
  • How does the performance in Tab 2 compare when the data is labeled using the best-performing tracker alone?
  • Since the architecture, loss function, and data are different than prior work, where do the observed performance gains come from?
Reviewer_3Tm6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses pseudo-label generation for the real-world adaptation of point trackers. The authors argue that frame-wise discrete selection among tracker predictions introduces temporal discontinuities and inherits the residual error of the selected candidate. To address this, they propose a candidate curator, which selects a reliable candidate, and coordinate recomposer, which uses the selected candidate as an anchor and predicts a residual correction. The generator is trained on Kubric with synthetically perturbed candidates, and the resulting pseudo-labels on RGBD1K are used to fine-tune several existing point trackers. Results show improved pseudo-label quality over the baselines and improved performance.

Critical strengths
  1. The paper identifies a concrete limitation of selection-based multi-tracker pseudo-labeling and quantifies it with the candidate switch rate and the switching jump, which is useful and well-motivated.
  2. Using the selected candidate as an anchor and predicting only a residual correction is a simple yet reasonable extension that can be easily combined with existing pipelines. 3 The generated pseudo-labels are applied to five different student trackers, showing generally consistent gains.
Critical weaknesses
  1. The main motivation of the paper is that discrete selection causes temporal discontinuities in pseudo-labels. However, temporal stability is measured only on the pseudo-labels themselves, not on the outputs of the fine-tuned students. Since student trackers have their own temporal modeling and are trained on a large number of trajectories, it remains unclear whether small frame-level jitter in pseudo-labels actually affects the trained model. It would strengthen the claim to report temporal stability metrics on the students' predictions. Furthermore, the switching jump is defined as the frame-to-frame displacement of the predicted coordinate, which also includes genuine object motion. A metric based on the change in error with respect to ground truth would be more appropriate.
  2. Limited isolation of the source of the gains in Table 2. Each tracker fine-tuned with the proposed pseudo-labels and adaptation strategy (RGBD1K, filtering, query curriculum, synthetic-real mixing) is compared against its original real-world fine-tuned version. As a result, it is difficult to disentangle whether the gains come from the proposed pseudo-labels or simply from the different training data and recipes. Experiments that change only the label source under the same fine-tuning pipeline appear to be necessary to validate the main contribution, and the paper would benefit from such controlled evidence.
  3. The improvements over the Verifier in Table 1 are marginal (mostly 0.1-0.6 points, with no change in avg on Kinetics), and no variance over multiple runs is reported, making it hard to judge the statistical significance of the gains. In addition, Track-On-R, the strongest student, performs worse on ITTO-MOSE after fine-tuning with the proposed labels with no explanation is given.
  4. Most of the analysis and ablations are conducted only on DAVIS, which contains only 30 videos. It would be more convincing to show the same trends on ITTO-MOSE, where the candidate distribution is much more challenging.
Questions for authors

There are typos, "Following . (Aydemir et al., 2026a)" (Sec. B.1, L706) and "self-attention(MHSA)" (Sec. 3.2, L249).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.