acceptodds
Under review as a conference paper at ICLR 2027

RECAST: Reusing Human Priors for Data-Scarce Animal 3D Pose estimation

Abstract

Monocular 3D animal pose estimation is inherently ambiguous, while the scarcity of 3D annotations makes reliable probabilistic 2D-to-3D lifting difficult to learn. We introduce RECAST, an efficient flow-matching framework that reuses human priors for data-scarce 3D animal pose estimation. RECAST employs a three-stage training strategy. First, probabilistic pretraining on abundant human poses learns a conditional 2D-to-3D flow that provides a transferable initialization for cross-species lifting. Second, low-rank feature and graph adaptation specializes the pretrained flow to animal morphology while retaining its conditional prediction structure. Third, skeletal refinement is introduced after adaptation has converged, further improving both pose accuracy and bone-length consistency. For efficient multi-hypothesis inference, we further propose Chordal Trajectory Branching (CTB). Inspired by the linear probability paths used in OT-CFM, CTB constructs intermediate hypothesis states through a closed-form approximation from a reference trajectory and integrates only their remaining trajectory suffixes. This avoids repeated prefix integration and substantially reduces redundant flow evaluations. RECAST achieves 29.86 mm and 18.67 mm P-MPJPE on Animal3D and CtrlAni3D, respectively. CTB matches standard multi-hypothesis inference within 0.01 mm while reducing function evaluations by 49%.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Animal 3D Pose Estimation, Cross-Species Flow Matching, Low-Rank Adaptation, Skeletal Refinement, Chordal Trajectory Branching

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_CLJ3OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper addresses 3D animal pose estimation under limited data. The proposed framework is human-to-animal transfer with LoRA, skeletal constraints including bone length supervision, and Chordal Trajectory Branching (CTB). In skeletal refinement, predefined species-specific bone lengths are used as the ground truths for regularization. CTB is used in inference phase to integrate multiple hypotheses. The proposed framework is evaluated on Animal3D and CtrlAni3D, showing comparative benefits in P-MPJPE.

Critical strengths
  • Human-to-animal transfer is a logical and useful direction for improved animal 3D pose estimation, and the Introduction section presents the motivation well.
  • In Section 3.3, the proposed method is interesting and seems reasonable for GCN adaptation across heterogeneous skeletons.
  • In Table 3, the benefit of skeletal supervision is presented and justified well.
Critical weaknesses
  • The benefit of the proposed method is not validated against a naive LoRA baseline. The backbone and flow matching losses largely follow FMPose3D, and Table 2 does not provide a baseline that is human-pretrained and naively finetune with LoRA (no GraphLoRA, no skeletal refinement, etc). To validate technical novelty, it would be better to compare with the naive LoRA baseline under the same parameter budget.
  • The proposed framework seems inefficient. While the parameter size of the original backbone is 16.3M, the propose method uses 4.4M LoRA parameters, which is roughly 27% of the original parameter size. As there is no train-val-test split available for hyperparameter tuning, there is a possibility that the model is overfit to the target benchmark.
  • The CTB ablation is required. The Figure 3 and Figure 9 do not ablate binary ablation (w/o CTB, with CTB).
  • Speed benefits presented in Figure 3 and Figure 9 do not convince actual usefulness. Vanilla multi-hypothesis inference could be parallelized in mini-batch. It would be more convincing to report the actual latency against non-CTB baselines.
  • The proposed framework might not properly utilize human prior. While FMPose3D uses 17 joints, the proposed framework uses 26 animal joints, which has 1.5x more joints compared to the source domain. Combined with the Weakness 2, it might indicate that the proposed method does not show comparative benefit over finetuning the entire backbone.
  • The Figure 8 is copy-pasted from the original FMPose3D paper without proper reference in the caption or in the main text.
Questions for authors
  • How much is the propose method beneficial against naive LoRA baselines under the same conditions (e.g., parameter budget)?
  • In Table 4, how does the best LoRA configuration change when train-val-test split is applied?
  • How does CTB affect the performance via ablation study (w/o CTB vs. CTB)?
  • What are the actual latencies of "w/o CTB" (no integration) and CTB?
  • Does the proposed framework show comparative efficiency or performance benefits over finetuning the entire backbone (finetune from the human-pretrained backbone)?
Reviewer_dYunOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper explores transfer learning for 3D animal pose estimation, adapting a model trained for 3D human pose estimation to the animal domain. The authors use FMPose3D (Wang et al., 2026)—which employs flow matching for 2D-to-3D pose estimation—as their baseline. Compared to the baseline, the key contributions of this paper are twofold:

  1. A LoRA-based approach to fine-tune a model trained on human pose data for animal pose estimation.

  2. Chordal trajectory branching (CTB) to reduce inference computation. The core idea is straightforward: instead of integrating from step 0 for a new pose, it uses a modified reference pose as the starting point.

Based on the weaknesses and questions outlined below, I believe the paper is not ready for publication in its current form and requires a thorough revision and deeper study.

Critical strengths
  1. The paper studies the underexplored problem of adapting 3D human pose estimation models to animal pose estimation. This is an interesting and valuable direction, as many domains outside of humans suffer from a scarcity of 3D pose training data.

  2. The evaluation validates CTB by showing that reducing the number of integration steps leads to only a minor increase in error. The insight behind CTB—that modifying the reference pose with a displacement linear to the initial displacement is effective—is also interesting.

Critical weaknesses
  1. My primary concern is the lack of quantitative comparisons with alternative transfer-learning approaches. In the main evaluation, the proposed approach benefits from additional human training data, whereas the baseline methods do not, making the comparison somewhat unfair. Since the core focus of the paper is domain adaptation rather than in-domain evaluation, the proposed method should be compared against other approaches that also leverage 3D human pose data for transfer.
  2. Regarding the LL-Lipschitz assumption in the appendix, there is no experimental analysis demonstrating whether or not this assumption holds in the learned model.
Questions for authors
  1. The multi-stage design lacks sufficient justification. Is a separate Stage III truly necessary? Could Stages II and III be combined into a single training stage using the composite loss in Equation 11?

  2. As noted on line 124, the authors mention "available supervision through pseudo-3D annotations and synthetic animal imagery." This raises a question: if existing solutions can provide pseudo-3D annotations, is the data-scarce problem still significant? Furthermore, how does this transfer-learning approach compare to simply synthesizing more pseudo-3D animal annotations for training?

  3. While CTB reduces inference steps during integration, different initial poses can be evaluated in parallel. Given that 2D-to-3D lifting is already computationally efficient, are the computational savings from CTB still significant in practice?

Minor Issues

  1. Line 109: There appears to be an incorrect reference in the related work section.
  2. Equation (6): Xb,jX_{b,j} and Xb,iX_{b,i} are not clearly defined in the text.
Reviewer_FSDmOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

RECAST addresses data-scarce 3D animal pose estimation from 2D joint coordinates by transferring a human-pretrained conditional flow model. Built on the FMPose3D backbone, it uses three training stages: human pose pretraining, animal-specific adaptation through feature-level LoRA and low-rank adjacency corrections, and post-convergence refinement with species-specific bone-length supervision. The paper also introduces Chordal Trajectory Branching (CTB), which approximates intermediate states of additional hypotheses using a reference trajectory and integrates only their remaining suffixes. The final prediction averages the generated hypotheses.

The method is evaluated on Animal3D and CtrlAni3D through comparisons with existing methods, training-stage ablations, adaptation-rank and inference-setting sweeps, and qualitative visualizations. The authors report P-MPJPE values of 29.86 mm and 18.67 mm, respectively. With 50 hypotheses and 20 integration steps, CTB reduces hypothesis-wise function evaluations from 1,000 to 510 while reportedly maintaining nearly unchanged prediction accuracy.

Critical strengths
  1. A coherent approach to cross-species adaptation under limited supervision. The framework explicitly accommodates skeletal differences through feature-level and graph-level adaptation, followed by bone-length refinement. Within-paper experiments provide encouraging evidence for the complete pipeline: weighted P-MPJPE decreases from 35.18 mm for animal-only training to 20.71 mm for the final model. The matched refinement comparison further isolates a benefit from skeletal supervision, improving P-MPJPE from 21.51 to 20.71 mm. These results support the practical value of the proposed training strategy.

  2. Simple inference-time reduction of multi-hypothesis computation. CTB approximates intermediate hypothesis states from a reference trajectory without additional training. Its computational saving is explicit: the default configuration reduces hypothesis-wise function evaluations from 1,000 to 510, while the reported averaged-prediction accuracy remains close to standard inference. Targeting redundant computation across hypotheses is a useful contribution that could make probabilistic pose estimation more computationally accessible.

Critical weaknesses
  1. Table 1 does not establish a comparison under equivalent input and inference conditions. HMR, WLDO, HMR2.0, and AniMer perform image-based reconstruction, whereas RECAST takes 2D joint coordinates as input. The manuscript does not specify whether its test-time coordinates are detected, manually annotated, or reprojected from 3D annotations. This distinction is consequential: annotation-derived inputs bypass the localization errors faced by image-based methods. Moreover, the cited FMPose3D animal results use one hypothesis and three integration steps, whereas RECAST defaults to 50 hypotheses and 20 steps with CTB. The reported differences therefore combine methodological changes with different inference budgets and potentially different test-time information. Table 1 should distinguish image-based reconstruction from conditional lifting, and provide comparisons using identical 2D inputs, evaluation protocols, and matched inference settings.

  2. The contribution of human pretraining is insufficiently isolated. Table 2 compares animal-only training with a pipeline combining human pretraining, low-rank adaptation, and subsequent skeletal refinement, but does not establish matched animal-training budgets, parameter-update policies, and objectives. Consequently, the improvement from 35.18 to 20.71 mm cannot be attributed specifically to human-prior transfer. Missing controls include human pretraining followed by conventional full fine-tuning, feature-only and graph-only adaptation, and an animal-only baseline receiving the same continued-training schedule and skeletal supervision. Table 3 appropriately controls continued training when assessing bone loss, but this comparison does not isolate the larger pretraining and adaptation gains. These omissions weaken the evidence for the paper’s central explanation of its improvements.

  3. The CTB accuracy-preservation claim is difficult to reconcile across figures and tables. At the default branching time of 0.5, Figure 3 appears to show approximately 29.88/18.65 mm on Animal3D/CtrlAni3D, whereas Table 3 reports 29.86/18.67 mm. Weighting the Figure 3 readings by the stated test-set sizes gives approximately 20.70 mm, compared with 20.71 mm in Table 2 and Figure 9. Although these discrepancies are small relative to the overall performance gains, they matter when the claimed accuracy difference is only 0.01 mm. The authors should provide a numerical table using the same checkpoint, test samples, and noise seeds, and specify which dataset or aggregate the 0.01 mm claim refers to. The 49% NFE reduction is arithmetically consistent; the precision of the associated accuracy claim requires clarification.

Questions for authors
  1. What is the source of the 2D inputs used for training and evaluation? For each dataset, please specify whether these coordinates are manually annotated, reprojected from the 3D annotations, or predicted by a detector. If a detector is used, please identify it and its training data. Are RECAST and FMPose3D evaluated using identical 2D coordinates and preprocessing? If annotation-derived coordinates are used, can you report results with detected keypoints and distinguish this lifting evaluation from the image-based reconstruction results in Table 1? Please also provide a comparison with FMPose3D under matched hypothesis counts and integration settings.

  2. How much of the improvement can be attributed specifically to human pretraining and the proposed adaptation strategy? Can you compare random initialization, human pretraining followed by full fine-tuning, and human pretraining followed by LoRA plus GraphLoRA, using the same animal data, training budget, skeletal supervision, and inference settings? Please report the training duration and parameter-update policy for each configuration, and clarify whether the animal-only baseline receives the same continued-training schedule and bone-length objective. Separate feature-only and graph-only adaptation results would further clarify the necessity of both components.

  3. Can you reconcile the default CTB results across Figure 3, Tables 2–3, and Figure 9? Figure 3 appears to show approximately 29.88/18.65 mm at branching time 0.5, whereas Table 3 reports 29.86/18.67 mm. Please provide unrounded, per-dataset and sample-weighted results for standard inference and CTB using the same checkpoint, test samples, and initial noise states. Which metric and evaluation population does the claimed 0.01 mm difference refer to? Were different runs or evaluation settings used across these figures and tables?

  4. Limited methodological novelty beyond closely related work. RECAST inherits much of its central formulation from FMPose3D: conditional flow matching for 2D-to-3D pose lifting, the Graph-Transformer backbone, linear interpolation-based velocity supervision, and multi-hypothesis generation through different noise initializations. FMPose3D already evaluates this framework on both human and animal pose benchmarks. Consequently, the contribution should be assessed primarily as an adaptation and inference extension of this existing framework, rather than as a new probabilistic formulation for animal pose estimation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.