- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a new object-centric point tracking framework. CEFFYL reformulates category-specific (horse) point tracking as correspondence in a canonical rest-pose space. CEFFYL is an effective extension of existing canonical amodal shape modeling to pairwise, category-specific point tracking, rather than a fundamentally new correspondence representation.
Critical strengths
Strong quantitative performance. CEFFYL consistently outperforms conventional point trackers, feature correspondence methods, and the closely related DualPM method across the three horse datasets.
Critical weaknesses
Limited novelty. The changes over DualPM are a pair-wise input, aggregation tokens and a descriptor head. Canonical-surface-mapping approaches to dense correspondence are not discussed: CSM, DensePose/CSE, DensePose for animals, and ViSER/LASR/BANMo. “Temporal attention” is two-frame interaction only, with no multi-frame temporal modelling.
Questions for authors
While CEFFYL demonstrates substantial improvements over DualPM, its main representational foundation, namely canonical amodal pointmaps, is inherited from prior work. The primary methodological additions appear to be pairwise temporal feature aggregation and descriptor-based correspondence refinement. Could the authors clarify whether the performance gains arise primarily from improved canonical shape estimation or from stronger cross-frame feature interactions? In particular, a matched pairwise correspondence baseline without explicit canonical 3D supervision, together with quantitative canonical consistency analysis, would help isolate the distinctive contribution of geometry-based modeling.
Rebuttal to Reviewer 9ibF
We would firstly like to thank reviewer 9ibF for reading and reviewing our work. We hope that the following responses will help clarify those points. We will update the paper by the 18th with all of the updates mentioned so that it is available for the final review. We respond early to initiate discussion.
Weaknesses (1) - Limited novelty in changes over DualPM
The changes over DualPM do include pairwise input, aggregation tokens and a descriptor head. It should also be mentioned that our architecture is transformer based, whilst DualPM is a U-Net architecture. These changes are fundamental to enable point tracking reliably (e.g. we boost performance by +7.9 on Horse10 in the main table). We also showcase the efficacy of these contributions in the ablations. Specifically, table 3.b in the main paper shows the contribution of each module on tracking performance and table 3.a shows that this is not simply down to the data that we use for training.
Weaknesses (2) - Canonical-Surface-Mapping approaches not discussed
We appreciate the reviewer’s concern. These approaches indeed share our goal - dense correspondence via a canonical space - they reach it through a fixed, category-level surface/mesh. CSM/A-CSM and DensePose/CSE regress each pixel to a single pre-defined template mesh per class, whilst LASR/ViSER/BANMo recover the surface by per-video test-time optimisation. We agree that these should be added to the related works section as they are relevant for context. DualPM's contribution over these works, as cited there, was to drop the template and predict an amodal pointmap to represent the deformation. Numerically, in the DualPM paper, results show large improvements over A-CSM in PCK and Chamfer distance. We thus do not compare to these works and focus on comparing to the stronger baseline of DualPM.
For completion, we compare to DensePose-CSE model under our protocol (SAM3 detection, identical metric, the strongest released animal model with the horse mesh and the best of four ROI configurations). The results are as follows:
Weaknesses (3) - Temporal attention is 2 frames only with no multi-frame modelling
To answer this, we also ran trained a variant of CEFFYL which accepts 4 frames (one query and 3 target). The results for these models are below:
As can be seen, the results are roughly the same. Increasing the number of frames does not seem to change performance much. Importantly, whilst the 2-frame original model takes ~12 hours on one H100 GPU, the 4-frame takes ~22 hours on 2 H100 GPUs. We were unable to extend this to 8 and 16-frame models due to our resource constraints.
Questions (1) - Contribution of 3D modelling?
The reviewer points out that it is hard to ascertain whether the performance gains arise primarily from improved canonical shape estimation or from the strong cross-frame interactions inside the model. In order to address this we train a new version of CEFFYL which simply predicts the descriptors instead of the 3D canonical shape. This is trained for the same 50k steps and uses the infoNCE loss described in the paper. Tracking is then calculated using the cosine similarity across descriptors. The results are as follows:
As can be seen, the addition of the 3D canonical shape modelling is critical for the tracking capability and shows that it contributes significantly to the final performance over the temporal aggregation.
The reviewer also mentions a "quantitative canonical consistency analysis". For this we measure the L2 error of the predicted amodal, canonical pointmap for 11 unseen sequences of VarenPoser during training. For comparison, we also include 11 random sequences from the training sequences. The results are:
Once again, we would like to thank the reviewer for all the effort they have put into reading and reviewing our work. We greatly appreciate it and if they have any further questions, we would be happy to answer them.