acceptodds
Withdrawn from ICLR 2027

Animal World: A Cross-Species Dataset for Social Animal Pose Understanding

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

20 views

Abstract

Animal pose understanding in the wild remains challenging due to large morphological variation, frequent occlusion, and complex social interactions. Reliable 2D pose estimation is not only fundamental to animal image understanding, but also provides an important source of supervision and evaluation for monocular 3D animal reconstruction, where accurate real-world 3D ground truth remains difficult to obtain. Existing animal pose benchmarks, however, are typically limited to particular species, sparse pose definitions, or isolated individuals, leaving robustness of animal pose estimation under diverse natural scenes insufficiently characterised. We introduce Animal World, a scene-centric dataset and benchmark for cross-species 2D animal pose understanding that jointly captures morphological diversity and inter-animal interaction in natural environments. Animal World provides a unified anatomical keypoint taxonomy and large-scale instance-level pose annotations, which enable consistent evaluation across species and socially complex scenes. Building on Animal World, we establish a comprehensive diagnostic benchmark that systematically evaluates the generalisation and robustness of animal pose estimation under species and domain shifts, occlusion, and diverse social configurations. Our extensive evaluations reveal substantial performance variation across species, domains, and interaction contexts, highlighting challenges beyond conventional single-animal benchmarks. We further show that Animal World supervision provides more informative 2D pose cues for downstream 3D animal reconstruction, demonstrating its value beyond 2D pose estimation alone. Animal World provides a unified data and evaluation foundation for advancing robust cross-species animal understanding in complex natural environments.

Primary area:
datasets and benchmarks
Keywords:
cross-species animal pose estimation, animal pose dataset, multi-animal scenes, social interaction

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_cVGSOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces Animal World, a "scene-centric" cross-species 2D animal pose dataset with 10,036 images and 19,132 instances, reportedly covering 124 species under a unified 30-keypoint taxonomy, with SAM3-derived and manually verified masks and boxes, and Qwen3-VL-8B-generated behaviour descriptions and image-level group-interaction labels. Images are sparsely sampled frames from more than 500 YouTube wildlife videos. Four human-pose top-down backbones (HRNet-W48, HRFormer-B, ViTPose-B, RTMPose) are benchmarked in MMPose with ground-truth boxes, followed by diagnostic analyses by occlusion level, social scenes, juveniles, rare species and instance count, plus cross-family, cross-dataset (APTv2), long-tail and pHash analyses in the appendix. Keypoints from a ViTPose trained on Animal World are then used as prompts for PRIMA post-optimisation and yield a small 3D reconstruction gain.

Critical strengths

1. The dataset addresses a genuine gap. Existing animal pose benchmarks such as AP-10K, APT-36K and Animal Kingdom contain few multi-animal or cross-species scenes and offer limited coverage of juveniles and endangered species. The proposed dataset targets these cases directly, and its 30-keypoint annotation is denser than the 17-keypoint schemes in common use.

2. The experimental analysis is organised around the properties the dataset is meant to capture. Rather than reporting a single aggregate score, the paper breaks results down by occlusion level, social context, age group and species rarity, and examines transfer to and from an existing dataset. This structure is appropriate for a dataset paper, although, as discussed below, the interpretation of several of these analyses depends on the annotation protocol and on the ground-truth-box evaluation setting.

3. The paper examines downstream utility for 3D reconstruction. The post-optimisation experiment with PRIMA (Table 8) is a reasonable attempt to show that the 2D annotations have value beyond 2D pose estimation, which is not commonly done in dataset papers of this kind.

Critical weaknesses

1. The central claim of the dataset concerns multi-animal and social-interaction scenes, but the evaluation protocol does not test this setting. According to Sec. 4, all baselines are evaluated conditioned on ground-truth bounding boxes, which the Limitations section also acknowledges. The four baselines are top-down methods that operate on a single cropped instance at a time. Under this protocol, the aspects that distinguish multi-animal scenes from single-animal ones, namely detecting all individuals, associating keypoints with the correct individual, and resolving instances whose boxes overlap substantially, are not evaluated. CrowdPose, which the paper cites as the human counterpart for crowded scenes, evaluates on full images without ground-truth boxes, and adopting an analogous protocol here would be natural. Pose estimators that operate on full images, as well as multi-animal tools such as multi-animal DeepLabCut and SLEAP, are available for this purpose. Given that the dataset already provides boxes and masks, a full-image evaluation appears feasible, and it would be needed to support the paper's claims regarding social and multi-animal scenes.

2. The annotation protocol is not described, which makes the reliability of the ground truth difficult to assess, in particular for the virtual surface points and for occluded keypoints. The annotation process is described only as "manually annotated by experts". Three aspects require clarification. (a) Several of the 30 keypoints (anterior, middle and posterior back, abdomen midpoint, tail midpoint, left and right upper forelimb) are not anatomical joints but reference points on a smooth body surface. Their placement is inherently subjective even when fully visible, and no inter-annotator agreement is reported. (b) The treatment of occluded keypoints is not specified: it is unclear whether annotators inferred the positions of invisible joints and whether such inferred coordinates are included in the AP computation. The occlusion analysis in Sec. 4.2 depends on this choice. If inferred coordinates are scored, label noise increases with the degree of occlusion, and the finding that models trained on Animal World perform better at low visibility may partly reflect agreement with the annotators' inferences; if they are not scored, the analysis measures robustness only on the remaining visible keypoints. (c) COCO-style AP depends on per-keypoint OKS sigmas, which are not reported for the 30 keypoints. Stating them would make the reported AP values easier to interpret and to compare with other datasets.

3. The benchmark includes only four generic human-pose backbones; no method developed for animal pose estimation, cross-species transfer or multi-animal scenes is evaluated. HRNet, HRFormer, ViTPose and RTMPose were designed for human pose estimation or generic dense prediction and are trained from scratch on Animal World. The animal pose literature offers natural candidates, such as the cross-species pretrained SuperAnimal models, the prompt-based CLAMP, the open-vocabulary keypoint detector X-Pose, and multi-animal tools such as DeepLabCut and SLEAP, but none of them is included. In addition, all diagnostic analyses are conducted with a single model (ViTPose-B). As a result, the paper does not show how existing animal pose methods perform on this data, and the evidence that the dataset poses challenges beyond those of existing benchmarks rests on comparisons among the authors' own models on their own split. Including animal-specific baselines would strengthen this argument considerably.

Questions for authors
  1. Please clarify how occluded keypoints were annotated. When a joint is not visible, were annotators asked to infer its position, and if so, what quality-control procedures (annotation guidelines, redundant annotation, inter-annotator agreement) were in place for these inferred labels? Are inferred coordinates included in the AP computation?
  2. Which OKS sigma values were used for the 30 keypoints, and how were they determined? Could the authors report inter-annotator agreement, particularly for the virtual surface points?
  3. The central claim of the dataset concerns multi-animal and social-interaction scenes, yet all reported results are conditioned on ground-truth bounding boxes, so no model is ever required to localise the animals or to associate joints with the correct individual. Could the authors provide an evaluation on full images without ground-truth boxes, so that the multi-animal setting the dataset is designed for is assessed directly?
  4. Could the authors include methods developed specifically for animal pose estimation in the benchmark, in addition to the human-pose backbones currently evaluated?
Reviewer_WqKPOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Animal World, a large-scale, scene-centric dataset and benchmark for cross-species animal pose estimation in natural environments. The dataset comprises 10,036 images and 19,132 annotated animal instances spanning 124 species, featuring a unified 30-keypoint anatomical taxonomy, instance-level segmentation masks, and bounding boxes. The authors establish a comprehensive benchmark using four representative pose estimation architectures and systematically evaluate model robustness and generalization across different species, datasets, and scene configurations. Experimental results demonstrate the importance of interaction-rich, age-diverse, and rare-species training data for improving pose estimation performance under challenging conditions. Furthermore, the paper demonstrates that dense 2D keypoint supervision from Animal World can improve downstream 3D animal reconstruction.

Critical strengths
  1. Animal World explicitly incorporates intra-species interactions, cross-species coexistence, age diversity, and rare species.
  2. The paper evaluates four representative pose estimation architectures and investigates several important factors, including occlusion, social interactions, age diversity, rare species, and cross-dataset generalization.
Critical weaknesses
  1. Insufficient details on annotation quality control and manual verification. In Section 3.2, the authors state that “Each instance … SAM3-derived … which are manually verified …”. However, the annotation quality control procedure remains unclear. First, the author does not specify any quantitative criteria for identifying failed or inaccurate SAM3-generated annotations, nor does it report annotation failure rates or correction statistics. Second, although the authors mention manual verification, they do not describe the verification protocol, the qualifications or expertise of the annotators, or how annotation consistency was ensured. While the paper states that the 30 keypoints were manually annotated by experts, it remains unclear whether these experts were also responsible for verifying the segmentation masks and bounding boxes. Providing these details would strengthen the reliability and reproducibility of the proposed dataset.
  2. Also the same problem in using Qwen3VL-8B. The authors do not describe the prompting strategy, annotation protocol, or whether any human verification was conducted
  3. In the Line 195, the authors state that Animal World is constructed from more than 500 wildlife videos collected from YouTube with frames sparsely sampled. However, it’s unclear that frames originating from the same source video can appear in both training and validation sets. Since frames from the same video may share highly similar backgrounds, animal identities, and visual characteristics. Such overlap could lead to overly optimized results that do not reflect generalization to unseen scenes.
  4. The contribution of dense 30-keypoint annotations to downstream 3D reconstruction is not adequately isolated. Section 4.7 and Table 8 compare PRIMA using keypoint predictions from ViTPose models trained on APTv2 and Animal World. The authors attribute the improvements to the richer anatomical representation provided by the 30-keypoint annotations. However, these two pose estimators differ in both training datasets and keypoint definitions.
Questions for authors

See Weaknesses. The main questions are as follows:

  1. Could the authors clarify the annotation verification protocol, including the qualifications of the annotators, the criteria used to identify incorrect annotations, and the proportion of annotations requiring manual correction?
  2. Could the authors clarify whether frames originating from the same source video are allowed to appear in both training and validation sets?
Reviewer_8YJwOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Animal World, a large scale dataset and benchmark for cross species 2D animal pose estimation in natural scenes. The dataset contains 10,036 images and 19,132 annotated animal instances covering 124 species, with a unified 30 keypoint taxonomy, instance masks, and bounding boxes. A major focus is on multi-animal and socially complex scenes, including intra-species groups, cross species co-occurrence, age diversity, rare species, and social interactions. The paper evaluates several pose estimation models and studies robustness under occlusion, cross dataset transfer, cross family generalization, social interaction, age variation, rare species, and increasing numbers of animals. It also evaluates the use of Animal World keypoints as supervision for 3D animal reconstruction.

Critical strengths
  1. The dataset addresses a meaningful gap in existing cross species animal pose benchmarks. Animal World places multi-animal and interaction rich scenes at the center of the benchmark rather than treating them as incidental examples. Around half of the images contain multiple animals, and the dataset also includes cross species combinations, juveniles, rare species, and social interactions.

  2. The evaluation makes good use of the structure of the dataset. The paper reports separate analyses for social scenes, age diversity, rare species, occlusion, animal count, cross dataset transfer, and cross family generalization. Several of these analyses show targeted effects. Social scene supervision provides much larger gains on interaction heavy subsets than on single animal images, while rare species supervision mainly improves performance on rare taxa.

Critical weaknesses
  1. The scene centric multi animal framing is not fully supported by the evaluation protocol. All pose baselines are conditioned on ground truth bounding boxes, which removes animal detection and much of the instance association difficulty emphasized in the motivation. The current experiments therefore evaluate keypoint localization under complex multi animal scenes, but do not evaluate the full multi animal perception problem suggested by the paper’s framing. The claims should be narrowed accordingly.

  2. The quality control of the social interaction annotations is not sufficiently documented. The paper uses Qwen3VL to generate behavior descriptions and image level group interaction labels, but does not state whether these annotations were manually verified or systematically audited. Since social interaction is a central dataset attribute and is used to define one of the main evaluation subsets, the reliability of these labels needs to be established more clearly.

Questions for authors
  1. Were the Qwen3VL generated behavior descriptions and image level social interaction labels manually verified? If so, how many annotations were checked, and what criterion was used to determine whether a generated label was correct?

  2. Are the training and validation sets disjoint at the source video level? If not, approximately how many source videos contribute frames to both splits? The current description only specifies an image level split and a perceptual near duplicate check.

  3. For the w/o Social, Social only, w/o Age, Age only, and w/o Rare training settings, how many training images and annotated instances are used in each setting? This would help determine how much of the reported difference is associated with data composition versus the total amount of supervision.

  4. In the occlusion evaluation, how is keypoint visibility defined? In particular, are out of frame or truncated keypoints treated the same as keypoints that are invisible because of inter animal or environmental occlusion? This distinction affects the interpretation of the visible keypoint count as a proxy for occlusion severity.

  5. In Table 8, ViTPose Base is trained on APTv2 while ViTPose AW is trained on Animal World, so both the training data and the keypoint annotation scheme differ. Is the improvement in this experiment intended to support the benefit of the 30 keypoint annotation specifically, or the benefit of Animal World supervision more generally?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.