acceptodds
Under review as a conference paper at ICLR 2027

CEFFYL: Modelling Deformable Object Shape for Improved Point Tracking

Abstract

Point tracking algorithms have consistently struggled with long-term and complex motion of objects. We introduce CEFFYL, a shape modelling method that utilises Canonical shape Embeddings and Feature Filtering to Yield Long-term point tracks. Specifically, we model the amodal representation of each pixel as a set of points on the rest pose of the object category. Once this mapping is established, correspondence and point tracking is estimated from this shared canonical space. We additionally learn a richer descriptor feature for local correspondences within the canonical space. We train CEFFYL on synthetic data of horses, particularly difficult objects to track due to their deformable nature and complex leg motion at speed. CEFFYL outperforms state-of-the-art point tracker AllTracker by and points on Horse10 and long Showground videos, respectively. We also outperform the single-frame DualPM shape model, the most recent work of this type, increasing performance by and points on the same metrics.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
point tracking, object modelling, 3D reconstruction, tracking

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_9ibFOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a new object-centric point tracking framework. CEFFYL reformulates category-specific (horse) point tracking as correspondence in a canonical rest-pose space. CEFFYL is an effective extension of existing canonical amodal shape modeling to pairwise, category-specific point tracking, rather than a fundamentally new correspondence representation.

Critical strengths

Strong quantitative performance. CEFFYL consistently outperforms conventional point trackers, feature correspondence methods, and the closely related DualPM method across the three horse datasets.

Critical weaknesses

Limited novelty. The changes over DualPM are a pair-wise input, aggregation tokens and a descriptor head. Canonical-surface-mapping approaches to dense correspondence are not discussed: CSM, DensePose/CSE, DensePose for animals, and ViSER/LASR/BANMo. “Temporal attention” is two-frame interaction only, with no multi-frame temporal modelling.

Questions for authors

While CEFFYL demonstrates substantial improvements over DualPM, its main representational foundation, namely canonical amodal pointmaps, is inherited from prior work. The primary methodological additions appear to be pairwise temporal feature aggregation and descriptor-based correspondence refinement. Could the authors clarify whether the performance gains arise primarily from improved canonical shape estimation or from stronger cross-frame feature interactions? In particular, a matched pairwise correspondence baseline without explicit canonical 3D supervision, together with quantitative canonical consistency analysis, would help isolate the distinctive contribution of geometry-based modeling.

1 comment
AuthorsOpenReview ↗

Rebuttal to Reviewer 9ibF

We would firstly like to thank reviewer 9ibF for reading and reviewing our work. We hope that the following responses will help clarify those points. We will update the paper by the 18th with all of the updates mentioned so that it is available for the final review. We respond early to initiate discussion.

Weaknesses (1) - Limited novelty in changes over DualPM

The changes over DualPM do include pairwise input, aggregation tokens and a descriptor head. It should also be mentioned that our architecture is transformer based, whilst DualPM is a U-Net architecture. These changes are fundamental to enable point tracking reliably (e.g. we boost performance by +7.9 on Horse10 in the main table). We also showcase the efficacy of these contributions in the ablations. Specifically, table 3.b in the main paper shows the contribution of each module on tracking performance and table 3.a shows that this is not simply down to the data that we use for training.

Weaknesses (2) - Canonical-Surface-Mapping approaches not discussed

We appreciate the reviewer’s concern. These approaches indeed share our goal - dense correspondence via a canonical space - they reach it through a fixed, category-level surface/mesh. CSM/A-CSM and DensePose/CSE regress each pixel to a single pre-defined template mesh per class, whilst LASR/ViSER/BANMo recover the surface by per-video test-time optimisation. We agree that these should be added to the related works section as they are relevant for context. DualPM's contribution over these works, as cited there, was to drop the template and predict an amodal pointmap to represent the deformation. Numerically, in the DualPM paper, results show large improvements over A-CSM in PCK and Chamfer distance. We thus do not compare to these works and focus on comparing to the stronger baseline of DualPM.

For completion, we compare to DensePose-CSE model under our protocol (SAM3 detection, identical metric, the strongest released animal model with the horse mesh and the best of four ROI configurations). The results are as follows:

ModelHorse10SG Horse
DensePose-CSE37.334.1
DensePose-CSE + SAM338.842.2
CEFFYL63.262.3

Weaknesses (3) - Temporal attention is 2 frames only with no multi-frame modelling

To answer this, we also ran trained a variant of CEFFYL which accepts 4 frames (one query and 3 target). The results for these models are below:

Input FramesHorse10SG Horse
2 (original)63.262.3
4 (new)63.062.6

As can be seen, the results are roughly the same. Increasing the number of frames does not seem to change performance much. Importantly, whilst the 2-frame original model takes ~12 hours on one H100 GPU, the 4-frame takes ~22 hours on 2 H100 GPUs. We were unable to extend this to 8 and 16-frame models due to our resource constraints.

Questions (1) - Contribution of 3D modelling?

The reviewer points out that it is hard to ascertain whether the performance gains arise primarily from improved canonical shape estimation or from the strong cross-frame interactions inside the model. In order to address this we train a new version of CEFFYL which simply predicts the descriptors instead of the 3D canonical shape. This is trained for the same 50k steps and uses the infoNCE loss described in the paper. Tracking is then calculated using the cosine similarity across descriptors. The results are as follows:

ModelHorse10SG Horse
CEFFYL (3D + descriptors)63.262.3
CEFFYL (descriptors only)44.550.1

As can be seen, the addition of the 3D canonical shape modelling is critical for the tracking capability and shows that it contributes significantly to the final performance over the temporal aggregation.

The reviewer also mentions a "quantitative canonical consistency analysis". For this we measure the L2 error of the predicted amodal, canonical pointmap for 11 unseen sequences of VarenPoser during training. For comparison, we also include 11 random sequences from the training sequences. The results are:

LayerSeen (Error in cm)Unseen (error in cm)
0 (visible)1.96.3
13.47.3
27.610.4
38.510.9

Once again, we would like to thank the reviewer for all the effort they have put into reading and reviewing our work. We greatly appreciate it and if they have any further questions, we would be happy to answer them.

Reviewer_d2t1OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents CEFFYL, a transformer-based method for long-term object-centric point tracking. It models each object pixel’s amodal representation in a shared canonical shape space, where correspondences are estimated through neighborhood search. And the authors introduce Showground, a benchmark of nine sparsely annotated, challenging horse-show videos.

Critical strengths
  1. The paper offers a clear formulation of object-centric point tracking as canonical shape modeling. CEFFYL maps object pixels to a shared canonical space, enabling correspondence estimation and long-term tracking.

  2. The paper introduces Showground, a new benchmark comprising nine sparsely annotated, long and challenging horse-show videos, providing a valuable testbed for evaluating long-term point tracking.

Critical weaknesses
  1. Annotations are provided by a single annotator, with no estimates of annotation consistency or error. The paper also does not specify how the evaluation frames are selected.
  2. All quantitative evaluations focus on horses, while results for other animals are limited to qualitative correspondences (Fig. 6). Thus, the benchmark and method primarily support a category-specific application, and generalization to a broader range of deformable objects remains an open question.
  3. A video of length TT requires T−1T-1 full pairwise forward passes, and the query appears to be re-encoded at every step.
Questions for authors
  1. Could the authors provide per-video results for all nine clips, clarify the criteria for selecting evaluation frames, and explain how the annotator handles point locations during occlusion?
  2. Can the query features and point map be cached, or can multiple target frames be processed jointly in a single forward pass?
2 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer d2t1 (PART 1)

We would like to thank reviewer d2t1 for taking the time to review our paper in detail. We appreciate the concerns they have raised and we have addressed them as best as we can. It should also be mentioned that we will update the PDF by the 18th so that it is available for the final round of reviewing. We offer this response early to encourage discussion.

Please be aware that we have split the rebuttal into multiple parts in order to deal with the character limit.

Weaknesses (1) - Robustness and Extending the Showground Benchmark

In the initial version we used a single annotator and selected 10 evaluation frames equidistant across the video, with manual edits for frames that were heavily occluded and therefore hard to annotate. However, during the review period, we densified the Showground benchmark, with annotations on every 5th frame. In order to combat the robustness issue, we chose to have one annotator do initial labelling, before a second annotator reviewed the annotations and corrected any errors. Below is a table outlining the expansion of the benchmark:

VideosEval. FramesAvg. Frames per VideoPoints (vis/occ/oov)Avg. Points per Video
Before9901041497 / 101 / 126.8
After10210810575804 / 7747 / 2736.8

The density increased the challenge of the benchmark. CEFFYL achieved 70.2 on the old and 62.4 on the and new. Importantly, the improvement over previous baselines remains consistent (+16.7 vs. +15.3 over AllTracker+SAM3 on old and new). Table 1 will be updated in the upload on the 18th.

We present further extension to this Showground benchmark that addresses also weakness (2).

Weaknesses (2) - All quantitative evaluations focus on horses...generalisation to a broader range of deformable objects remains an open question

We additionally collected two more evaluation benchmarks of 10 dog video and 10 cat videos, annotated in the same way as the densified Showground for horses. The statistics for each of these are as follows:

VideosEval. FramesAvg. Frames per VideoPoints (vis/occ/oov)Avg. Points per Video
SG Dog1019319714955 / 8308 / 10987.4
SG Cat1013336715458 / 2851 / 14587.3

We then trained our CEFFYL model using synthetically generated dog and cat data. We used publicly available meshes and posed them according to reference photos for each breed using pre-trained key point detectors. Sequences were then created by interpolating between two randomly chosen poses and the pointmaps in the same way as we did for horses (details will be added to supp). We were able to train a dog, cat and joint (horse+dog+cat) variant of CEFFYL:

ModelHorse10SG HorseSG DogSG Cat
AllTracker49.038.642.436.0
AllTracker + SAM354.347.151.236.1
CEFFYL-horse63.2 (+ 8.9)62.3 (+15.2)55.3 (+ 4.1)36.5 (+0.4)
CEFFYL-dog52.8 (- 1.5)55.8 (+ 8.7)64.6 (+13.4)50.3 (+14.2)
CEFFYL-cat47.3 (- 7.0)54.2 (+ 7.1)60.9 (+ 9.7)51.7 (+15.6)
CEFFYL-joint64.5 (+10.2)62.5 (+15.4)65.5 (+14.3)55.6 (+19.5)

All of our models out-compete AllTracker and AllTracker+SAM3 across each benchmark (+X in table). Training a joint model performs best, showcasing the generalisability of our method. Will include qualitative results showcasing the extended Showground benchmark and our results in paper.

AuthorsOpenReview ↗

Rebuttal to Reviewer d2t1 (PART 2)

Weaknesses (3) - A video of length T requires T-1 fuel pairwise forward passes

We outline the inference cost in figure 7 of the supplementary. The correspondence models tested underperform our model in tracking accuracy whilst also being slower during inference. The reconstruction models tested although slightly faster during inference are far below in terms of tracking accuracy. Point trackers, such as AllTracker and CoTracker3, do not perform 3D object modelling, so are faster but trade this for accuracy. DualPM is slower, larger and less accurate.

If we compute the DINOv2 features of the query image and cache it (instead of recomputing for each pair), this would increase inference speed by ~8%.

Questions (1) - Could the authors provide per-video results for all nine clips...?

As Showground has now been extended, the results for each of the videos are as follows:

Video NumberCEFFYLAllTracker + SAM3AllTracker
158.932.324.3
263.351.941.4
363.048.77.7
460.952.052.5
554.229.729.3
657.242.543.3
750.849.144.6
868.966.056.7
970.156.955.4
1075.441.430.6
Mean62.347.138.6

The reviewers also asked for clarity on how evaluation frames were chosen and how the annotators handled occlusions. Evaluation frames are every 5th frame, where the query is the first frame of the video. For occluded points, the annotators mark the points as occluded and we asked them to place the points where they believed the point is if they could see it.

Questions (2) - Can the query features and pointmaps be cached? Or can multiple target frames be processed jointly?

We address the idea of caching in weakness 3. We trained another version of CEFFYL that takes a query frame and 3 target frames. These results show that performance is roughly the same as the 2-frame model. However, whilst the 2-frame model takes ~12 hours on 1 GPU, the 4-frame model requires ~22 hours across 2 GPUs, greatly increasing training compute and runtime. The inference time of the 4-frame model is 36% lower.

Input FramesHorse10SG Horse
2 (original)63.262.3
4 (new)63.062.6

We would like to thank the reviewer again for their thoughtful feedback and hope that the rebuttal helps with their final reviewing round. If there are any further questions, then we would be happy to answer them.

Reviewer_uRcBOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes CEFFYL, a category-specific point tracker that reformulates long-term point tracking as dense mapping to a canonical shape. Given a query and a target frame cropped with SAM3 masks, a transformer on frozen DINOv2 features predicts, for every pixel, an amodal pointmap in the rest-pose space of a horse template. Unlike a standard pointmap, which stores only the visible surface point, the amodal pointmap stores every intersection of the pixel's ray with the object surface, so self-occluded points are also represented. Since each surface point maps to a pose-invariant canonical coordinate, tracking reduces to nearest-neighbour search in canonical space, followed by re-ranking with learned descriptors. Occlusion is inferred from the amodal layer at which the match is found. The model is trained purely on synthetic horses (VarenPoser), with ground truth obtained by ray casting posed meshes and mapping intersections back to the rest pose. The paper also introduces Showground, a benchmark of 9 long (701 to 1961 frames) horse show videos. CEFFYL reports the best d_avg, OA, and AJ on Horse10, PFERD, and Showground against point trackers, correspondence models, 4D reconstruction methods, and DualPM.

Critical strengths

Pose-invariant identity for long-term correspondence, with explicit occlusion handling. Mapping each pixel to a fixed coordinate on a category template gives every surface point an identity that is invariant to articulation and viewpoint, so matching does not depend on local appearance remaining similar across frames. Combined with the amodal representation, occluded points retain a predicted location and visibility is obtained from the matched layer rather than estimated from vanishing appearance cues. This is a principled alternative to cost-volume-based tracking for highly articulated objects, and the occlusion metrics support it: CEFFYL improves OA over AllTracker by 10.0 and 11.7 points on Horse10 and PFERD, and AJ substantially on all three benchmarks (Table 2).

Critical weaknesses

1. Training is purely synthetic, but the sim-to-real gap is not analyzed, and the source of real-world robustness is unclear. All benchmarks are real, so the results show the method works despite the gap, but not how much is lost or what bridges it. Table 3a suggests the synthetic data itself is problematic: fine-tuning AllTracker on VarenPoser reduces Showground d_avg from 53.5 to 41.9, and retraining DualPM on VarenPoser lowers performance on both benchmarks (55.3 to 52.3 on Horse10, 61.7 to 55.6 on Showground). That CEFFYL nevertheless performs well on the same data suggests that the frozen DINOv2 backbone may be largely responsible for robustness to real images, but the paper compares only backbone sizes (Table 6a), not frozen versus fine-tuned features. It is also not stated whether VarenPoser covers the extreme poses (jumping, galloping) that dominate Showground.

2. It is unclear how much of the gain comes from 3D shape modelling rather than from direct query-to-target matching and mask-constrained search, compounded by a specialist versus generalist comparison. Most baselines are generic trackers, while CEFFYL is trained specifically on horses. On the long Showground videos, untrained DINOv2 nearest-neighbour matching with SAM3 masks reaches 65.9, well above AllTracker+SAM3 (53.5) and only 4.3 points below CEFFYL (70.2). The spatial-attention-only variant of CEFFYL (62.7, Table 3b) and DualPM (61.7) are both below this simple baseline on Showground. This suggests that much of the headline long-term improvement may stem from matching each frame directly to the query (avoiding drift accumulation) and from restricting the search to the object mask, rather than from canonical shape modelling. The gap to DINOv2 NN+SAM3 is much larger on Horse10 (63.2 vs. 45.6) and PFERD (84.0 vs. 69.7), so the contribution of shape modelling appears substantial in some settings but is not isolated by any controlled experiment.

Questions for authors

Clarifications:

  1. Was DINOv2 kept frozen throughout, and have you compared against fine-tuning it? This would clarify whether the backbone, rather than the canonical formulation, accounts for robustness to real images.
  2. What is the pose distribution of VarenPoser, in particular how many frames contain jumping or galloping, and how does this relate to Showground?
  3. Since the method models 3D shape, why is evaluation restricted to 2D metrics (d_avg, OA, AJ)? Is any evaluation of canonical or amodal geometry accuracy possible, for instance on held-out synthetic data or on PFERD with fitted poses?

Requested analyses that could change my assessment:

  1. A controlled baseline that shares CEFFYL's backbone, SAM3 crops, and direct query-to-target matching but matches on features instead of predicted canonical coordinates. This would isolate the contribution of shape modelling from direct matching and mask-constrained search.
  2. A breakdown of performance by motion type or difficulty (e.g., leg crossings, self-occlusion, jumps). In which situations does the template-based formulation provide the largest advantage over existing trackers?
  3. A sim-to-real analysis, such as comparing performance on held-out synthetic sequences with the real benchmarks.

Suggestion for future work:

  1. Adapting to real video with pseudo-labels or self-training, as has proven effective in recent point tracking work, would test whether the remaining sim-to-real gap limits performance.
2 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer uRcB (PART 1)

We would like to thank reviewer uRcB for their thorough and thoughtful review. We will update the main PDF with these new results before the 18th, so that they are available during the final review stage. We opted for an early response to initiate discussion.

Please be aware, we have split the following rebuttal into several parts in order to deal with the character limit.

In the first weakness, the reviewer shared concerns about synthetic-only training. We follow the norm for point trackers, where they all train on synthetic (simulated) data. Previous works, such as PIPs (Harley et al., ECCV 2022), PointOdyssey (Zheng et al., ICCV 2023), TAPIR (Doersch et al. ICCV 2023), TAPTRv3 (Qu et al., ICLR 2026) and CoTracker (Karaev et al., ECCV 2024), have found point trackers trained solely on simulated data to be highly competitive and directly applicable to real data without adjustments for the sim-to-real gap. AllTracker (Harley et al., ICCV 2025), is almost exclusively trained with synthetic data (apart from the ~2% of driving optical flow data). Furthermore, the DualPM model that we take inspiration from also trains exclusively on synthetic data.

Below is a list of responses for each point (Questions for Authors 1-7 as these also cover the remaining concerns):

1) Was DINOv2 kept frozen throughout, and have you compared against fine-tuning?

The reviewer rightly points out that we do not include an ablation on whether fine-tuning DINOv2 changes performance. We had opted to comply with prior works (like DualPM and VGGT) where DINOv2 was frozen. In order to fully understand this effect, we trained CEFFYL with the DINOv2 unfrozen and also with LoRA in order to see the difference:

ModelHorse10Showground
CEFFYL63.262.3
CEFFYL (DINOv2 unfrozen)61.445.3
CEFFYL (LoRA fine-tuned)62.358.7

Un-freezing the DINOv2 weights or using LoRA does not provide much advantage to a frozen backbone. This is likely the result of the sim-to-real gap noted by the reviewer or the limited size of the training set compared to the original training of DINOv2. This also confirms that it is not the backbone that is the reason for CEFFYL’s strong performance.

In the second weakness, the reviewer also mentions that "untrained DINOv2...only 4.3 points below CEFFYL" on showground, which is a smaller gap than other benchmarks. This is to do with the significant variation in horse sizes in Showground. Specifically, the average crop box in Showground is 10.3% of the image compared to 32.2% in Horse10. As we use the same δavg\delta_{avg} metric, this is easier for DINOv2 as the error is penalised less. To show this we present the analysis below:

CoverageDINOv2CEFFYLGap
0-1%71.872.7+0.9
1-2%64.966.5+1.6
2-4%54.959.5+4.6
4-8%49.456.8+7.4
8-16%40.149.6+9.5
>16%32.140.4+8.3

These results show a significantly bigger gap for larger horses.

2) What is the pose distribution of VarenPoser?

The poses in VarenPoser cover horses walking, trotting and turning. There are only a few examples of rearing up and no examples of jumps. As no rearing labels exist, we measure this specifically by looking at the pitch of the horse's body against the horizontal. The results are as follows (0.4 degrees is the median pitch):

Body PitchTraining Frames
> 30 degrees0.41%
> 40 degrees0.10%
> 50 degrees0.03%

This shows that rearing is a rare occurrence in VarenPoser. However, the dataset does include random camera variations meaning the model sees horses from different poses and articulations.

AuthorsOpenReview ↗

Rebuttal to Reviewer uRcB (PART 2)

3) & 6) Why are evaluations restricted to 2D metrics? Canonical and tracking accuracy on held-out synthetic sequences?

It is possible to evaluate the performance of the shape model compared to ground-truth points on training and held-out synthetic data. We perform amodal canonical reconstruction and point tracking evaluations on the 11 sequences that were not used in training our model (we use 1160 sequences for training). These sequences include 120 frames each. We compare that to 11 sequences from the VarenPoser dataset that were seen during training.

LayerSeen (Error in cm)Unseen (error in cm)
0 (visible)1.96.3
13.47.3
27.610.4
38.510.9

This evaluates the canonical space. It is understandably easier to get a low error on the training set, particularly for the visible layer. If reviewer uRcB has other suggestions for a metric in 3D, we’d be happy to add that.

Additionally, we compare tracking accuracy on the same seen and unseen sequences for 256 query points per sequence:

d_avg
Seen73.3
Unseen72.0

It is clear from the two analyses that performance drops slightly between seen and unseen sequences, as is expected. Tracking accuracy is also slightly higher for synthetic sequences when compared to real videos (refer to Horse10 and Showground results under (1)). This is expected as the model as real videos introduce additional challenges of appearance and occlusion.

4) Isolating the contribution of shape modelling

We appreciate the need for this ablation. We trained a version of CEFFYL with only the descriptors, removing the canonical shape modelling but keeping everything else intact (backbone, SAM3 crops, direct query-to-target matching). This means that no 3D shape is modelled, CEFFYL simply outputs an amodal feature representation for each point. Tracking is then completed using cosine similarity. The results can be seen below:

ModelHorse10Showground
CEFFYL (3D + descriptors)63.262.3
CEFFYL (descriptors only)44.550.6

This ablation confirms the efficacy of including the 3D modelling of the canonical, over simply matching through features. This also shows that our strong results are not due to direct query-to-target matching even when using the same backbone and mask constraining. We hope this rebuttal removes the concern.

5) - Breakdown of performance by motion types?

As noted, VarenPoser has varying poses but does not include jumping. The reviewer was interested in breaking down the tracking accuracy over different motion types, in order to see where the benefit lies for our model. We conduct two experiments. The first shows accuracy on points positioned on the legs vs. the body on Horse10:

Motion TypeCEFFYLAllTracker + SAM3Gap
All63.254.7+8.5
Legs63.348.0+15.3
Body62.962.8+0.1

This shows that a large portion of our boost over AllTracker on Horse10 is due to better understanding of leg motion. It is also important to point out that we do not lose performance for points on the body, which AllTracker finds easy to track.

We also measure tracking accuracy for jumping and non-jumping frames in Showground. Specifically, we get claude to identify the segments where jumps occur in our benchmark and then manually confirm these segments. The results are as follows:

Motion TypeCEFFYLAllTracker + SAM3Gap
All62.447.1+15.3
Jumps59.744.7+15.0
Non-jumps63.347.7+15.6

Performance is consistent across diverse motions, including the more challenging (for both AllTracker and CEFFYL) jumping sequences.

Once again, we thank the reviewer for their careful consideration of our work. If there are any further questions, we would be happy to answer them.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response (to all reviewers)

We would once again like to thank all of the reviewers for their thoughtful and thorough reviews of our work. The questions and concerns they raise are much appreciated and we hope that our individual rebuttal responses below address these concerns. The PDF has been updated to match these responses too and all updates can be found in blue throughout the paper and supplementary.

The following is a list of highlights from our individual rebuttal responses (found underneath each review):

  1. Showground Benchmark - We extend this benchmark to include three subsets, each with 10 videos, on challenging, long-term videos of horses, cats and dogs, annotated by two hired annotators. Statistics of these benchmarks have been added to the paper and allow us to better quantitatively assess our models for generalisation to other animals. Overall we annotate a total of 5372 frames across 30 videos (10 for each animal) with 37952 point annotations (with labels of visible, occluded or out-of-view).
  2. Extended Ablations - We offer additional ablations on fine-tuning DINOv2 during training, removing the 3D modelling to show the contribution this makes, increasing the number of input frames to the model and assessing tracking accuracy across different motion types (body vs. legs and jumping vs. non-jumping).
  3. Other animals and generalisability - We show results for training additional dog-specific, cat-specific and joint models on newly generated data during the review period. We show that each category-specific model achieves the best performance on their respective subset of Showground. Furthermore, our joint model (trained on the synthetic horse, dog and cat data) is shown to be best overall, further supporting our claims for generalisation of our methodology.
  4. Sim-to-Real Gap and Evaluation - We also expand our evaluations to include canonical, amodal reconstruction accuracy on a held-out synthetic test set of VarenPoser, as well as tracking, as requested by multiple reviewers, in order to better clarify the sim-to-real gap as our model is trained using synthetic data.
  5. Expansion of Related Works - As rightly pointed out by one reviewer, we also expand the related works to include canonical-surface-mapping models such as CSM, DensePose-CSE, LASR, BANMo and ViSER. We also add results for DensePose-CSE to the main table for comparison.
  6. Inference Cost - We further clarify the inference cost due to the pairwise formulation, comparing to other models that we evaluate against in the main paper. We also include details on how much inference can be sped up by caching DINOv2 features for the query image.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.