acceptodds
Under review as a conference paper at ICLR 2027

CAST: Real-Time Monocular Motion Capture and Animation for Any Skeleton Topology

Abstract

This paper studies the problem of recovering motion from monocular video, even when the target skeleton topology differs from the subject observed in the video. Although previous methods have made efforts to resolve this problem, they remain vulnerable to joint-rotation errors that are coupled across the skeleton and can propagate along its kinematic hierarchy. To mitigate these errors, we present CAST, a topology-aware motion capture framework that combines learned local rotations with geometric correction guided by predicted global bone directions. Our observability analysis characterizes which rotational components can be constrained by outgoing bone directions. Guided by this analysis, GRACE uses reliability-gated alignment to correct composed orientations in parent-to-child order, leaving unconstrained components to the learned rotation predictor. The motion predictor and correction module are trained jointly with motion-derived supervision, using a frozen image encoder. Experiments under the TopoCap evaluation protocol show that the proposed framework improves motion recovery across different target topologies, including unseen structures. Reference-based cross-skeleton evaluation on a curated Mixamo benchmark and a user preference study further assess transfer to different target structures. A causal CAST-B variant achieves over 51 FPS on a single NVIDIA A100 GPU, with timing covering image encoding and motion prediction. Code is available at https://cast-mocap.github.io/.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
motion capture, arbitrary skeletons, monocular video, cross-skeleton animation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_un5XOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CAST addresses monocular motion capture for arbitrary target skeletons. It combines learned local-rotation prediction with global bone-direction constraints and introduces GRACE, which corrects hierarchical rotation errors according to topology-dependent rotational observability and predicted directional reliability. Experiments cover matched-rig, unseen-topology, cross-skeleton, causal, and sparse-observation settings, with substantial improvements over the evaluated baselines.

Critical strengths
  1. Correcting local rotations with global bone-direction constraints offers a well-motivated approach to mitigating error propagation through the skeletal hierarchy.
  2. The method achieves consistent performance improvements in both matched-rig and cross-skeleton motion recovery while supporting real-time causal inference.
Critical weaknesses

The technical distinction from existing methods needs to be clarified. Adaptive HybrIK already addresses error accumulation through adaptive inverse kinematics, while TopoCap uses global orientation prediction. The paper should explain how GRACE differs from these approaches and what additional contribution its confidence weighting and learned gating provide.

Questions for authors
  1. How does GRACE differ from Adaptive HybrIK and TopoCap in handling error propagation? In particular, what do confidence weighting and learned gating add beyond these existing approaches?
  2. Could the authors ablate direction supervision, confidence weighting, learned gating, and hierarchical correction separately? Two useful controls would be to retain direction supervision without applying correction and to compare hierarchical correction with correction applied after full forward kinematics (FK).
  3. What input resolution, numerical precision, and compilation settings were used for each baseline? Were comparable optimizations applied to all methods? Since the methods use different temporal window lengths, please report online latency and offline throughput separately.
3 comments

Thank you for the constructive feedback and helpful suggestions. Below, we clarify the differences from prior methods, provide component ablations, and report updated inference timings with optimized baselines.

Q1. Differences from Adaptive HybrIK and TopoCap. Adaptive HybrIK recovers rotations from predicted human joints through adaptive inverse kinematics, addressing bone-length mismatch with a body template. Our target skeletons have fixed bone lengths, and GRACE addresses rotation errors accumulated during FK. TopoCap first predicts global rotations, then recovers each child's local rotation by multiplying the inverse of its parent's global rotation by the child's global rotation. This conversion couples rotation prediction errors across parent–child relationships. GRACE instead starts from predicted local rotations and uses independently predicted global bone directions to correct composed orientations during FK. Each corrected orientation is passed to the next level.

The learned gate limits interference from inaccurate bone-direction predictions early in training. Confidence weights account for bone-specific occlusion and skeletal structure, giving more reliable directions greater influence. Ablations are reported in Q2.

Q2. Component ablations. We evaluate six CAST-B DINOv3 variants. CAST-B w/o GRACE removes direction supervision and correction. Direction-only retains direction supervision without correction. Uniform weighting gives valid directions equal solver weights, retaining the learned gate. Fixed gate sets g=1g=1, retaining predicted confidence weights. Post-FK applies corrections after a complete, uncorrected FK pass. Full GRACE corrects during FK, propagating corrected orientations to children. Entries report MPJPE / PA-MPJPE (mm, lower is better).

ConfigurationZooMObjaverse seenMObjaverse unseen
CAST-B w/o GRACE50.41 / 38.7863.71 / 36.8786.64 / 50.71
Direction supervision only47.82 / 36.9863.96 / 37.5985.00 / 49.69
Uniform alignment weights43.24 / 33.2960.75 / 34.9879.36 / 46.58
Fixed gate g=1g=143.01 / 33.6259.32 / 35.6278.60 / 47.98
Post-FK correction43.64 / 34.1959.42 / 34.9977.49 / 45.95
Full GRACE42.68 / 33.2355.39 / 32.9279.21 / 46.52

Full GRACE performs best on Zoo and seen, while Post-FK performs best on unseen.

Q3. Baseline optimization and updated timings. Original Table 5 used official TopoCap and MCA V2 implementations without additional inference optimization. We now optimize both baselines.

DINO, T5 where used, and motion networks use BF16 with FlashAttention acceleration. CAST and MCA V2 use torch.compile. TopoCap is faster uncompiled. MCA V2 processes only actual joints, eliminating default padding to 150 joints.

We remeasured all methods on another NVIDIA A100-SXM4-80GB machine with PyTorch 2.7.0+cu128. CAST's timing setup is unchanged, with minor runtime variations between machines. TopoCap and MCA V2 use 224 × 224 inputs with DINOv3 and DINOv2, respectively. CAST uses 256 × 256 with DINOv3. Timing includes image encoding and motion prediction.

Each entry reports time (ms) / FPS. Offline time is per non-overlapping window of the listed length, with FPS = 1000 × Frames / Time. Causal time is per incoming frame, with online FPS = 1000 / Time.

MethodFrames30 joints52 joints89 joints
TopoCap (10 steps)8876.08 / 9.13924.02 / 8.66972.57 / 8.23
TopoCap (300 steps)824812.39 / 0.3224735.94 / 0.3225231.61 / 0.32
MCA V23254.13 / 591.2056.26 / 568.8354.49 / 587.28
CAST-B81135.25 / 598.90150.70 / 537.49159.70 / 507.20
CAST-L81174.53 / 464.09185.02 / 437.80201.14 / 402.71
CAST-B (causal)122.16 / 45.1321.39 / 46.7622.27 / 44.91

We will update Table 5 values and supplement the appendix with baseline timing details.

I thank the authors for their response, which has addressed my concerns. I have no further questions.

Thank you for your follow-up and for confirming that our response has addressed your concerns.

In the revised manuscript, we will expand Table 3 with additional component ablations and update Table 5 with remeasured inference times using the optimized baseline implementations. The corresponding implementation and timing details, including input resolutions, numerical precision, and compilation settings, will be provided in the appendix.

Thank you again for your time and constructive feedback.

Reviewer_RQHiOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a new framework for unified motion capture from the monocular video. Through modelling the correlation between bone-direction constraints and rotation recoverability, the proposed method significantly reduce the estimation error. This paper also propose a real-time streaming pipeline for motion application scenarios.

Critical strengths
  1. The proposed method is interesting. In the literature, MocapAnything V1 uses classical FK method and MocapAnything V2 replaces it with learnable neural networks and achieves a better results. In this paper, the authors discussed a more well-designed optimization procedure and achieves significant improvement on both matched-rig setting and cross-skeleton setting.
  2. The real-time demo is cool, which enables many downstream applications.
  3. The overall qualitative results are satisfying.
Critical weaknesses
  1. The presentation is supposed to be improved. In the methodology part, it lacks some intuitive illustration about why we use this formula and why other kind of formation are not such suitable for the current scene.
  2. The authors should report the inference time for different methods.
  3. The proposed GRACE contains many detailed technical designs. However, there are few ablation study about these choices, such as the value of hyper-parameters and whether gated correction is necessary.
Questions for authors
  1. The visualization quality of TopoCap and MocapAnything V2 are far from their official report. Is there any reason for it? As for MocapAnything V2, how about directly using the original weights?

  2. I'm curious about how different solvers influence the performance. For example, using different solvers may lead to different inference time and optimization accuracy. The authors may report results on more different solvers.

3 comments

Thank you for your constructive feedback.

Q1. Baseline visualization discrepancies and evaluation of the original MCA V2 weights.

TopoCap. We evaluated TopoCap using its officially released checkpoint, which appears to have been trained only on MObj. On MObj, we obtain slightly better results than reported, but Zoo performance is poor, so we leave the corresponding Table 1 entries blank.

MCA V2. MCA V2’s Zoo split and 1,000-sample MObj subset were unreleased during our experiments, so we used a random Zoo split and TopoCap’s MObj splits. Consequently, our Zoo test set may overlap with the checkpoint’s training data, while the full MObj test set includes out-of-distribution cases beyond its subset. For a fair comparison, we retrained MCA V2 on CAST’s training splits. Original checkpoint results are shown below, with reproduced MCA V2* results in parentheses. Lower values are better for all metrics.

DatasetMPJPEPA-MPJPEMPJVECD
Zoo85.59 (105.52)49.71 (67.04)8.98 (10.35)0.062 (0.077)
MObj seen469.26 (94.38)120.19 (58.62)26.30 (16.71)0.219 (0.071)
MObj unseen441.51 (103.04)108.46 (61.98)24.46 (16.44)0.215 (0.074)

Q2. Solver comparison: reconstruction accuracy and runtime.

Using CAST-B, we compared GRACE’s SVD-based rotation-alignment solver with a quaternion eigensolver and 10-iteration SO(3) gradient optimization. Solver-only runtime was measured for a 30-joint, 81-frame input at batch size 1 on an NVIDIA A100 80GB GPU in FP32 eager mode. Lower MPJPE and solver time are better.

SolverZoo MPJPEMObj seen MPJPEMObj unseen MPJPESolver time
SVD (ours)41.6853.5279.221.26
Quaternion eigensolver41.6853.5279.221.78
SO(3), 10 iterations41.6553.5378.9130.70

MPJPE varies by less than 0.32 mm (0.4%) across solvers on all splits, so accuracy is largely insensitive to solver choice. SVD has the lowest runtime in our implementation.

W3. Component ablations. Following suggestions from several reviewers, we conducted a more detailed ablation study. All configurations use CAST-B with DINOv3. CAST-B w/o GRACE removes direction supervision and correction. Direction-only retains direction supervision without correction. Uniform weighting gives valid directions equal solver weights, retaining the learned gate. Fixed gate sets g=1g=1, retaining predicted confidence weights. Post-FK applies corrections after a complete, uncorrected FK pass. Full GRACE corrects during FK, propagating corrected orientations to children. Entries report MPJPE / PA-MPJPE (mm, lower is better).

ConfigurationZooMObj seenMObj unseen
CAST-B w/o GRACE50.41 / 38.7863.71 / 36.8786.64 / 50.71
Direction supervision only47.82 / 36.9863.96 / 37.5985.00 / 49.69
Uniform alignment weights43.24 / 33.2960.75 / 34.9879.36 / 46.58
Fixed gate g=1g=143.01 / 33.6259.32 / 35.6278.60 / 47.98
Post-FK correction43.64 / 34.1959.42 / 34.9977.49 / 45.95
Full GRACE42.68 / 33.2355.39 / 32.9279.21 / 46.52

W2. Inference time. In the original Table 5, TopoCap and MCA V2 were evaluated using their official implementations without additional inference-time optimization. Following Reviewer un5X’s suggestion, we optimized the inference implementations of both baselines and remeasured all methods using PyTorch 2.7.0+cu128 on an NVIDIA A100-SXM4-80GB GPU. For CAST, we kept the timing protocol unchanged, but its new timings differ slightly from the original ones because a different GPU was used.

DINO, T5 (where applicable), and the motion networks run in BF16 with FlashAttention acceleration. CAST and MCA V2 use torch.compile, while TopoCap runs uncompiled because this was faster in our tests. MCA V2 processes only the valid joints, avoiding the default padding to 150 joints. TopoCap and MCA V2 use 224 × 224 image inputs with DINOv3 and DINOv2, respectively, while CAST uses 256 × 256 inputs with DINOv3.

Each entry reports time (ms) / FPS. Offline time is per non-overlapping window of the listed length, with FPS = 1000 × Frames / Time. Causal time is per incoming frame, with online FPS = 1000 / Time.

MethodFrames30 joints52 joints89 joints
TopoCap (10 steps)8876.08 / 9.13924.02 / 8.66972.57 / 8.23
TopoCap (300 steps)824812.39 / 0.3224735.94 / 0.3225231.61 / 0.32
MCA V23254.13 / 591.2056.26 / 568.8354.49 / 587.28
CAST-B81135.25 / 598.90150.70 / 537.49159.70 / 507.20
CAST-L81174.53 / 464.09185.02 / 437.80201.14 / 402.71
CAST-B (causal)122.16 / 45.1321.39 / 46.7622.27 / 44.91

We will update Table 5 and supplement the appendix with baseline timing details.

Thanks for authors' response, which addresses most of my concerns. After reading other reviewers' comments and the corresponding rebuttal, I lean to acceptance.

Thank you for your positive follow-up and for taking the time to consider our response and the broader discussion. We are glad that these have addressed most of your concerns.

In the revised manuscript, we will expand Table 3 with additional component ablations and update Table 5 with remeasured inference times using optimized baseline implementations. The appendix will include the details for these two tables, including input resolutions, numerical precision, and compilation settings, as well as the solver comparison.

Thank you again for your time and constructive feedback.

Reviewer_iXkuOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

CAST recovers motion from monocular video for a supplied skeleton, using predicted bone directions and gated geometric corrections to improve learned joint rotations.

Critical strengths
  • The paper addresses a practical problem: recovering animation for different target rigs without explicit source–target joint correspondences.
  • The main novelty is the integration of topology-dependent, reliability-gated correction into learned motion prediction, with a clear account of which rotations bone directions can constrain.
  • The causal model exceeds 51 FPS on an A100 including image encoding, and the user study provides complementary evidence of animation quality (Tables 2 and 5).
Critical weaknesses
  • Since Table 3 adds direction supervision and correction together, could separate ablations show how much each contributes and whether learned reliability and gating outperform uniform weights and a fixed gate?

  • Position errors can miss axial twist and leaf rotations, could you report rotation errors before and after correction, grouped by tree depth and observability class?

  • CAST and MCA V2 use different architectures and temporal windows. does the performance gap remain with matched image features, window lengths, and model capacity?

  • Quantitative cross-skeleton evaluation covers only six Mixamo skeletons, could a small evaluation of transfers between more structurally different rigs test how broadly the gains hold?

Questions for authors

Which parts of CAST besides GRACE explain the sizable improvement over MCA V2, given that CAST without GRACE already performs substantially better?

2 comments

Thank you for your constructive feedback and recognition of our work.

Q1. CAST differs from MCA V2 in its backbone architecture.

(1) Decoder input. MCA V2 decodes rotations from joint positions and mixes ground-truth and predicted poses during training. CAST decodes rotations directly from video latent features, with position prediction serving only as auxiliary supervision.

(2) Skeleton Embedding. CAST replaces MCA V2's joint-name and reference-image conditioning with rest-pose geometry and topology, removing T5 embeddings.

(3) Block design. MCA V2's encoder applies joint self-attn, image cross-attn, then graph self-attn, without FFNs. CAST applies per-frame image cross-attn, graph self-attn, then per-joint temporal self-attn, each with its own FFN.

(4) Rotations output. MCA V2 directly predicts local rotations. CAST predicts rest-relative increments: $R_j = R_j^{\mathrm{rest}} \Delta R_j$..

W1. More detailed ablations.

Following suggestions from several reviewers, we conducted a more detailed ablation study. All configurations use CAST-B with DINOv3.

Direction-only retains supervision without correction. Uniform weighting replaces confidence weights with equal solver weights, retaining the learned gate. Fixed gate sets g=1g=1, retaining confidence weights. Post-FK applies correction after a complete, uncorrected FK pass. Entries report MPJPE / PA-MPJPE (mm, lower is better).

ConfigurationZooMObjaverse seenMObjaverse unseen
w/o GRACE50.41 / 38.7863.71 / 36.8786.64 / 50.71
Direction supervision only47.82 / 36.9863.96 / 37.5985.00 / 49.69
Uniform alignment weights43.24 / 33.2960.75 / 34.9879.36 / 46.58
Fixed gate g=1g=143.01 / 33.6259.32 / 35.6278.60 / 47.98
Post-FK correction43.64 / 34.1959.42 / 34.9977.49 / 45.95
Full GRACE42.68 / 33.2355.39 / 32.9279.21 / 46.52

Direction-only improves Zoo and unseen but not seen, while full GRACE reduces both errors on all three splits relative to direction-only. Learned confidence weights outperform uniform weights on both metrics across all splits. Learned gating improves both metrics on Zoo and seen and PA-MPJPE on unseen, although fixed g=1g=1 gives lower unseen MPJPE.

We will add these new ablation results to Table 3.

W2. Rotation errors by observability class.

We compare independently trained CAST-B DINOv3 models, reporting GD (rad) and L2Q for w/o GRACE → full GRACE (lower is better). GD measures local rotation error; L2Q measures global quaternion error.

Observability classZoo GDZoo L2QMobj seen GDMobj seen L2QMobj unseen GDMobj unseen L2Q
Rank 00.217 → 0.2171.30 → 1.220.277 → 0.2791.76 → 1.590.263 → 0.2612.00 → 1.89
Rank 20.275 → 0.2661.47 → 1.350.327 → 0.3162.10 → 1.910.340 → 0.3572.50 → 2.37
Rank 30.251 → 0.2520.38 → 0.360.274 → 0.2930.53 → 0.480.311 → 0.3810.70 → 0.70

L2Q improves in 8 of 9 group–split pairs, with only a slight increase for unseen rank 3 (<0.01). Local GD changes remain mixed, particularly on unseen. GRACE propagates corrected parent rotations to descendants through recursive FK, allowing even leaves to benefit in global rotation space without directly correcting their local rotations. This is consistent with the overall global-rotation gains across all three splits.

W3. Model size and inference time with 32-frame windows.

Our original timing used the released MCA V2 code without additional inference optimization. We now optimize MCA V2 using BF16 and torch.compile for DINOv2, T5 and the motion network. We also remove its default padding to 150 joints, processing only actual joints.

CAST-B and MCA V2 use comparably sized backbones: both have 8 encoder blocks and 8 decoder blocks, with a hidden width of 256 and 8 heads. We report 32-frame window inference times (ms) on another NVIDIA A100-SXM4-80GB, including image encoding and motion prediction. MCA V2 uses DINOv2 at 224 × 224, while CAST-B uses DINOv3 at 256 × 256, with both models producing 256 patch tokens per frame.

ModelFrames30 joints52 joints89 joints
MCA V23254.1356.2654.49
CAST-B3288.4690.2588.25

Thank you to the authors for their response. I have no further questions. Please ensure that the relevant ablation studies are improved accordingly. I maintain my score.

Reviewer_Ds2GOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies monocular motion capture for arbitrary target skeletons, including cases where the target skeleton differs from the subject in the video. It builds on a modified MoCapAnything V2 backbone and introduces GRACE, which uses predicted global bone directions to correct rotations along the kinematic tree. The paper analyzes which rotational degrees of freedom can be constrained by bone directions and reports improved positional accuracy on Zoo, MObjaverse, and Mixamo. It also demonstrates key-frame motion completion and real-time inference.

Critical strengths
  1. The motivation is clear, and using global bone directions to correct rotation errors along the skeleton is a reasonable idea.
  2. The position-based results are promising across multiple test sets, with consistent gains from GRACE.
  3. Real-time inference and Blender integration show potential for practical use.
Critical weaknesses
  1. Rotation correction is the main contribution, yet rotation-angle and angular-velocity errors are not reported. Better MPJPE does not necessarily mean better rotations or skinned animation. Rotation metrics with and without GRACE are particularly needed.
  2. The evaluation mainly uses rendered videos. Despite the live human demonstration, systematic comparisons on real humans and diverse animals are missing, leaving real-world generalization unclear.
  3. There is no comparison with SMPL-based methods on a standard human benchmark to clarify the accuracy–generality trade-off. This would be useful additional evidence; outperforming human-specific methods is not required.
Questions for authors
  1. Could you report rotation-angle and angular-velocity errors for CAST, MCA V2*, and TopoCap under a consistent protocol, including results with and without GRACE?
  2. Could you provide baseline comparisons on real videos of humans and multiple animal species, including challenging or failure cases? Please clarify the preprocessing and target-skeleton selection.
  3. Have you evaluated against SMPL-based methods on a standard real-human benchmark? This would be useful additional evidence of the accuracy–generality trade-off.

I currently lean toward weak rejection. The approach is well motivated, but evidence for rotation quality and real-world generalization is still missing. I would be open to weak acceptance if the authors provide convincing rotation metrics, rotation-level GRACE ablations, and real-video comparisons. A human-benchmark comparison would be helpful, but is not a prerequisite for changing my rating.

3 comments

Thank you for the constructive feedback and for recognizing the motivation and practical value of our method. We provide additional results on rotation quality, real animal images, and GRACE within a human-specific model below.

Q1. Rotation quality and GRACE ablation. We agree that MPJPE alone is insufficient to assess rotation quality. We therefore report two complementary metrics. GD measures the mean angular difference between predicted and ground-truth parent-relative joint rotations, with the root evaluated in the global frame. It assesses local rotation accuracy and is reported in radians. L2Q measures the L2 distance between predicted and ground-truth global joint quaternions, concatenated across joints after hierarchical rotation composition. This dimensionless metric evaluates the resulting global orientations. Each table entry is GD / L2Q, with lower values being better.

MethodZooMobj seenMobj unseen
TopoCap-0.653 / 5.140.642 / 5.55
MCA V2*0.298 / 3.200.466 / 7.220.397 / 7.97
CAST-B w/o GRACE0.255 / 2.080.309 / 2.880.314 / 3.37
CAST-B0.250 / 1.930.304 / 2.620.330 / 3.20

CAST-B achieves the best L2Q on all three splits and the best GD on Zoo and Mobj seen. Compared with CAST-B w/o GRACE, L2Q decreases by 7.0%, 8.8%, and 5.0%, respectively. These gains show improved global rotation reconstruction across both seen and unseen skeletons, consistent with GRACE's goal of correcting orientations during hierarchical composition. On Mobj unseen, the global improvement is accompanied by a small increase in local GD (0.314 to 0.330 rad), indicating a trade-off between local angular accuracy and global rotation reconstruction on this split.

We will add both GD and L2Q to Table 1.

Q2. Real animal evaluation. For quantitative evaluation on real animals, we identified Animal3D, which provides SMAL annotations for individual images. To accommodate both single-frame inputs and a target skeleton absent from our original training data, we fine-tuned CAST-L with LoRA for 500 steps on the Animal3D training split. We report the adapted model's results on all 320 test images spanning 40 animal categories, using the same normalization as in the paper.

MethodMPJPE (mm) ↓PA-MPJPE (mm) ↓GD (rad) ↓L2Q ↓
CAST-L + LoRA, 500 steps68.6540.910.2842.65

Qualitative results show that strong lighting, pronounced shadows, and low image resolution can reduce reconstruction accuracy. Moreover, Animal3D currently provides only single-frame images rather than video sequences, preventing the use of temporal cues to resolve ambiguities in monocular skeleton reconstruction. More visualizations are available at: https://console-solid-mini-baptist.trycloudflare.com/animal3d/

Our preprocessing code is available in https://github.com/CAST-mocap/CAST/tree/main/inference and the https://github.com/CAST-mocap/CAST/blob/main/preprocess/mixamo/scripts/prepare_mixamo.py. We use SAM 2.1 masks to crop the RGB frames, then resize them to 256 × 256 and normalize them. For the target character, we extract the joint hierarchy and rest-pose transforms from its FBX to construct CAST's skeleton inputs. The same RGB preprocessing was used for our human real-time demo. For Animal3D, we use the provided foreground masks and construct the 35-joint SMAL rest skeleton from the supplied individual shape parameters.

Q3. Evaluation with a human-specific model. We additionally integrate GRACE into SMPLer-X-S and compare it with the baseline on MTP. Both models are trained for 6,000 steps on the 3,187 training images and evaluated on all 544 validation images. We report MPVPE over the body and both hands after pelvis alignment and PA-MPVPE after similarity alignment, excluding only face vertices. The reference meshes are SMPL-X fits provided by MTP.

MethodMPVPE (mm) ↓PA-MPVPE (mm) ↓
SMPLer-X-S112.1578.59
SMPLer-X-S + GRACE107.4073.26

GRACE reduces MPVPE and PA-MPVPE by 4.24% and 6.78%, respectively, showing that its rotation correction also benefits mesh reconstruction within a human-specific model.

More visualizations are available at: https://console-solid-mini-baptist.trycloudflare.com/mtp/

Thanks for the detailed response and the extra results.

The new rotation metrics (GD / L2Q) and the rotation-level GRACE ablations address my main concern. I also looked through the visualization pages, and the results look reasonable to me. I understand that real-world videos have no ground truth, so a quantitative comparison there is hard.

I'm raising my score to 3 (weak accept). For the final version, it would still be nice to include some side-by-side qualitative comparisons with the baselines on real videos, and to add the new rotation metrics and ablations to the paper as promised.

Thank you for your positive reassessment and for supporting acceptance. We are glad that the additional rotation metrics and GRACE ablations addressed your main concern.

In the revised manuscript, we will add GD and L2Q to Table 1 in the main paper. The additional evaluations on real animals will be included in the appendix. Thank you again for your constructive feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.