acceptodds
Under review as a conference paper at ICLR 2027

OmniMM: A Unified Language-Motion Model with Biomechanical Skeleton Motion

21 views

Abstract

Recent progress in multimodal human motion modeling has enabled increasingly capable systems for generating and understanding motion, yet most operate in surface-driven parametric representations that are not designed to faithfully reflect human skeletal anatomy. We introduce **Omni-Motion Model (OmniMM)**, a unified framework that treats anatomy-driven biomechanical skeleton motion as a first-class modality alongside conventional parametric motion and language. OmniMM jointly models these modalities, enabling motion generation, motion understanding, bidirectional cross-representation fitting, and synchronized joint generation within a single model. To make this possible at scale, we construct paired parametric–biomechanics–language data by recovering biomechanical skeleton motion from existing SMPL sequences while preserving their original annotations. OmniMM models continuous motion representations with modality-specific Transformer towers coupled through shared attention, enabling specialized modeling while supporting cross-modal interaction. Experiments demonstrate effective modeling and translation across semantic, surface-driven, and anatomy-driven motion spaces, including the generation of parametric motion aligned with its corresponding biomechanical skeleton motion.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Biomechanical Skeleton Motion, Language Motion Model

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.67 over 3 reviews, above 99% of the others.
Reviewer_Hx14OpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper introduces OmniMM, a single model designed for handling language, SMPL-H motion, and biomechanical skeleton motion (BSM). This is achieved using a three-tower Mixture-of-Transformers that combines discrete text tokens with continuous tokens from two motion tokenizers, enabling text-conditioned generation, motion captioning, bidirectional conversion, and the simultaneous generation of both motion representations. The paired dataset is created by fitting BSM to synthetic markers placed on SMPL-H sequences from six datasets. The experiments compare generation and captioning performance with retrained baselines, assess fitting accuracy and speed, and investigate anatomical-limit violations in generated motion.

Critical strengths
  • The unified model, together with the paired dataset, enables language-conditioned motion modeling to be applied to biomechanical skeletons. OmniMM supports generation, understanding, and conversion between motion representations using 26,253 paired clips totaling 51 hours (Table 1). The generated BSM retains the anatomical benefits of the target representation: for the evaluated elbows and knees, violations exceeding 10 degrees occur in 0.01–0.53% of generated BSM frames, compared with 21.4–55.1% for the generated parametric stream (Table 6).
  • The experiments indicate useful improvements when compared against baseline models under controlled conditions. The learned baselines are retrained using their official implementations on the same representations and data splits (Appendix B.1). OmniMM achieves lower T2M/T2B FID than the compared baselines, although it does not lead on every retrieval metric (Table 3). Learned fitting takes 0.440/1.700 seconds for M2B/B2M, respectively, compared with 7.501/6.966 seconds for the 20-iteration optimization baseline (Table 5). B2M also improves accuracy, while M2B trades some accuracy for speed. The paper clearly states that joint generation does not impose the BSM stream’s anatomical constraints on the parametric stream.
Critical weaknesses
  • The evaluation reference and scoring procedure in Table 4 require clarification. Although M2T and B2T have separate modality-specific evaluators, they share the same GT row. Its MMDist of 0.966 matches neither GT value in Table 3, which reports 0.943 for parametric motion and 1.151 for biomechanical motion. If the same ground-truth pairs and embeddings are used, reversing the retrieval direction alone would not explain this difference in mean distance. MotionGPT3 also has unusually high MMDist values of 2.634/2.424 despite strong retrieval scores. These observations raise questions about the reference samples, embedding normalization, and scoring pipeline, rather than establishing that the results are incorrect.
  • Missing split details make the reported captioning gains difficult to assess. FineDance sequences are divided into windows with up to 50% overlap, and all subsequences of a dance receive the same two template captions. TRUMANS also uses template-based caption augmentation (Appendix B.4). The paper does not state whether subsequences from each original recording are kept within a single split or specify the test-set composition for Tables 3–4. Without this information, it is unclear whether closely related motion segments occur across splits or how strongly the captioning scores depend on repeated templates. R-precision is averaged over 20 evaluation replications, but no spread is reported, and variability across training runs is also unspecified.
Questions for authors
  • Which evaluator and reference samples produced the shared GT row in Table 4, and why does it differ from Table 3? Please specify the embedding normalization and distance measure used for MMDist and Diversity, and confirm that all methods used the same scoring and post-processing pipeline. The human-caption reference should also be distinguished from a strict upper bound on retrieval performance.
  • How were the training, validation, and test sets constructed for each source dataset? Were overlapping FineDance windows and other subsequences from the same original recording kept within the same split? Please provide the dataset composition and sample counts underlying Tables 3–4.
  • Generated BSM has fewer anatomical-limit violations than ground truth for three of the four evaluated joints. Could the author compare their range-of-motion distributions to check whether the tokenizer’s joint-limit loss suppresses valid motion near the limits? Please also specify which clips or generated samples were used for Tables 6–7.
  • Section 4 states that data construction follows the same surface-marker and fitting strategy as BioAMASS, but Table 2a reports lower marker errors, such as 1.08 versus 1.70 cm for CMU bony markers. Does this difference come from AddBiomechanics settings or versions, sequence filtering, or the error computation?
7 comments
AuthorsOpenReview ↗

R1: reference rows and scoring pipeline of Table 4

We thank the reviewer for the careful reading, which helped us identify two issues in Table 4. We address them below together with the questions in Q1. Neither issue changes the conclusions of the paper.

Scoring pipeline. All methods in Tables 3 and 4 are scored with the same test clips, the same 32-way R-precision over 20 replications, and the same post-processing. All methods also use the same frozen evaluators, except the MotionGPT3 rows of Table 4, which by our mistake were scored with an older version of our evaluator. We apologize for this error and have re-scored MotionGPT3 with the correct evaluator (see below), so that all methods are now evaluated identically.

Embedding normalization and distance. Each evaluator encodes a clip and a caption into 512-D embeddings, L2-normalizes both, and multiplies them by the evaluator's learned scale. The embedding-based evaluation metrics are computed using Euclidean distances:

  • R-precision ranks each clip against the 32 captions of its group by their distance.
  • MMDist is the distance between a clip and its own caption, averaged over all clips.
  • Diversity is the distance between two randomly drawn generated clips, averaged over 300 pairs.

Since the two evaluators have different scales, MMDist and Diversity should be compared across methods within one evaluator, not across the motion and biomechanics columns.

Clarification of GT reference scores. We thank the reviewer for spotting this. We reported only the parametric-motion GT scores in Table 4. Checking our evaluation process, we noticed a minor inconsistency in how the reference captions were chosen, which produced the slightly different GT numbers. Most test clips have several human captions, while the runs behind Table 3 use the first caption, whereas the run behind Table 4 draws one at random. The evaluator itself is deterministic, and re-scoring with the first caption reproduces Table 3:

GT clips scored againstR@1MMDist
First caption (Table 3)0.6120.943
Random caption (Table 4)0.5970.966
First caption, re-scored0.6120.943

The two reference MMDist values differ by only 0.023, a small gap compared with the differences between methods. This inconsistency affects only the printed reference rows and does not change the comparison between methods. The reviewer is also right that B2T needs its own reference, for which we now use the biomechanical GT scores already reported in Table 3.

Why MotionGPT3 has a high MMDist despite strong retrieval. We thank the reviewer for this observation. As noted above, the high MMDist comes from our mistake of scoring the MotionGPT3 rows with an older version of our evaluator. Re-scoring them with the correct evaluator gives:

TaskMethodR@1↑R@2↑R@3↑MMDist↓
M2TMotionGPT3 (submitted)0.6060.7460.8082.634
M2TMotionGPT3 (corrected)0.5480.7190.8050.989
M2TOmniMM0.6820.8320.8950.837
B2TMotionGPT3 (submitted)0.6870.8220.8762.424
B2TMotionGPT3 (corrected)0.6000.7610.8331.117
B2TOmniMM0.6900.8400.8991.009

With the corrected scores, MotionGPT3's MMDist is consistent with its retrieval accuracy, while OmniMM continues to outperform all baselines across all metrics on both tasks. Thus, the conclusions of Table 4 remain unchanged. We also correct the inconsistent decimal formatting in the MotionGPT3 rows.

Revision (Main paper). We now report one reference row per evaluator on a fixed caption per clip, and include the corrected MotionGPT3 rows (Table 4).

Revision (Appendix). We describe the human-caption rows as reference values in Appendix B.2.

AuthorsOpenReview ↗

R2: dataset splits and evaluation composition

We thank the reviewer for raising this concern. We address the dataset splits, the evaluation composition of Tables 3 and 4, the template captions, and the variability in turn, which also answers Q2.

Split construction and composition. For FineDance and TRUMANS, we split into training, validation and test sets at the level of the original recordings, before temporal segmentation. All windows derived from one recording therefore fall in the same split, and we verified that no recording or overlapping window is shared across splits. FineDance has 162 / 20 / 21 training / validation / test dances, and TRUMANS has 261 / 33 / 33 recordings. For AMASS, we use the official HumanML3D split without modification, following the standard protocol of prior text-to-motion and motion-to-text work. The remaining datasets use our own 8:1:1 training / validation / test splits. After the split, we filter out clips outside the valid length of 20 to 200 frames (1 to 10 s at 20 fps), which is the HumanML3D length filter. The table gives the number of test clips scored in Tables 3 and 4, and the number of clips used for training.

DatasetTraining clipsEvaluated clips (Tables 3–4)
AMASS8,4331,563
CombatMotion6,192703
100STYLE1,650206
FineDance2,776275
TRUMANS1,490188
HumanSC3D55060
Total21,0912,995

Template captions. FineDance uses two captions per dance style, and TRUMANS uses template-based augmentation with motion-specific descriptions. Although caption templates may be shared between training and test sets, the FineDance and TRUMANS test recordings are entirely unseen during training, and together they make up only a small portion of the evaluated clips.

Evaluation variability. Over the 20 evaluation replications, the standard deviation of R-precision lies between 0.002 and 0.008 for every method and experiment.

Training variability. Each complete multi-stage training run takes about 57 hours on four L40S GPUs (Appendix A.1.2), which makes multiple independent runs impractical within the rebuttal period. We acknowledge this limitation.

Revision (Main paper and Appendix). We have added these splitting procedures and dataset statistics to Appendix B.6 (Table 17), and the standard deviations to Tables 3 and 4.

AuthorsOpenReview ↗

Answers to Q1 and Q2

A1: evaluator reference and scoring details

We thank the reviewer for this question. All parts are addressed in R1, which identifies the reference samples behind each GT row, specifies the embedding normalization and distance used for MMDist and Diversity, confirms the shared scoring pipeline, and distinguishes the human-caption reference from an upper bound.

A2: split construction and dataset composition

We thank the reviewer for this question. All parts are addressed in R2, which describes how each source dataset is split, confirms that all windows of a FineDance or TRUMANS recording stay within one split, and gives the per-dataset composition and sample counts underlying Tables 3 and 4.

Reviewer_Edb5OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces OmniMM, a unified framework for modeling language, surface-driven parametric motion, and anatomy-driven biomechanical skeleton motion. The authors construct paired data by generating synthetic markers from existing SMPL sequences and fitting biomechanical skeletons, obtaining 26,253 clips (51 hours) with 152,935 text annotations. OmniMM uses separate continuous motion tokenizers and modality-specific Transformer branches with shared attention. Motion tokens are generated through flow matching, while language is modeled autoregressively. The framework supports seven tasks, including text-conditioned generation, motion understanding, bidirectional fitting, and synchronized joint generation. Experiments evaluate these capabilities and examine anatomical differences between the two motion representations.

Critical strengths
  1. By extending existing marker-based fitting techniques to six motion domains, the authors construct 26,253 paired motion clips with 152,935 text annotations. Although the underlying fitting techniques are not new, this dataset helps address the scarcity of paired biomechanical data and provides useful supervision for cross-representation and anatomy-aware motion-language modeling.
  2. OmniMM supports seven tasks within a single framework, showing promising performance in motion understanding and cross-representation fitting. For M2B, it achieves an MPJPE of 0.080 m versus 0.078 m for the optimization-based baseline, while reducing the reported runtime from 7.501 s to 0.440 s. These results demonstrate the potential of learned cross-representation fitting without per-sequence optimization.
Critical weaknesses
  1. Insufficient evidence for the benefits of unified modeling over simpler alternatives. The paper lacks a direct comparison between synchronized joint generation (T2BM) and a simpler cascaded T2M + M2B pipeline. Although Table 3 compares standalone and joint generation, it does not establish whether synchronized generation improves motion quality, cross-representation consistency, or efficiency over sequential generation and fitting. Ablations of key architectural components are also missing. Consequently, while unified modeling is demonstrated to be feasible, its advantages over simpler alternatives remain insufficiently established.
  2. In Table 5, the optimization-based baselines and OmniMM perform different native-representation conversion tasks. For M2B, the baseline fits SKEL to SMPL joints, whereas OmniMM predicts 57-D BSM motion. For B2M, the baseline recovers SMPL joint positions through SKEL fitting and mesh-based regression, while OmniMM predicts 77-D SMPL-H motion. Although evaluated in a common target joint space within each direction, the metrics mainly measure positional alignment, and the runtimes reflect different procedures. Thus, the results do not fully establish advantages in equivalent native-representation conversion tasks.
Questions for authors
  1. Have the authors compared T2BM with a cascaded T2M + M2B pipeline? How do they differ in generation quality, cross-representation consistency, and inference time? This comparison would help establish whether synchronized joint generation offers meaningful advantages over a simpler alternative.
  2. In Table 5, how do differences in native prediction targets and computational procedures affect the accuracy and runtime comparisons? Could the authors clarify whether these results demonstrate advantages in full representation conversion or primarily in joint-space alignment?
4 comments
AuthorsOpenReview ↗

R1: advantages of unified modeling over simpler alternatives

We thank the reviewer for this suggestion. We address the two parts of the comment in turn.

We first clarify that the paper evaluates synchronized generation through Table 3 and Table 5 jointly, not through Table 3 alone. The two tables measure complementary aspects of synchronized generation:

  • Table 3 (per-stream quality). T2BM preserves the quality of both output streams relative to the same model's standalone T2M and T2B tasks. That means joint generation does not degrade either stream.
  • Table 5 (cross-representation consistency and efficiency). The two jointly generated streams achieve 0.090 m MPJPE and 0.055 m PA-MPJPE in the common joint space. Separately, OmniMM's learned M2B/B2M fitting is substantially faster than per-sequence optimization.

Comparison with cascaded pipelines. Following the reviewer's suggestion, we compare our synchronized joint generation (OmniMM T2BM) against cascaded T2M + M2B pipelines and their B2M counterparts. Each cascade combines a text-to-motion generator, either OmniMM or T2M-GPT (trained for parametric and biomechanical motion generation), with a fitting step. Among the evaluated models, only OmniMM supports learned M2B/B2M fitting. Using the same OmniMM checkpoint for generation and fitting also enables a controlled comparison between synchronized and sequential generation:

  • OmniMM T2M → OmniMM M2B and OmniMM T2B → OmniMM B2M
  • T2M-GPT → OmniMM M2B and T2M-GPT → OmniMM B2M

All pipelines follow the evaluation protocols of Tables 3 and 5. Each cascade is evaluated on its final output stream. Runtime is measured per clip and includes generation and learned fitting, excluding joint-space evaluation procedures. All model components use the same checkpoints as reported in the paper.

PipelineParam. FID↓Param. R@1↑Param. R@3↑Biomech. FID↓Biomech. R@1↑Biomech. R@3↑Pair MPJPE / PA-MPJPE (mm)↓Time (s)↓
OmniMM T2BM (ours)0.1110.4790.7440.0620.4950.74789.9 / 55.34.40
OmniMM T2M → M2B–––0.1320.3860.639142.3 / 82.93.23
OmniMM T2B → B2M0.2460.3460.586–––122.5 / 83.93.23
T2M-GPT → OmniMM M2B–––0.2310.4690.71682.8 / 55.81.12
T2M-GPT → OmniMM B2M0.2480.3650.615–––91.3 / 57.31.06
  • Generation quality. T2BM achieves the best generation quality in both representations. Compared with OmniMM cascades, joint generation improves biomechanical FID from 0.132 to 0.062 and parametric FID from 0.246 to 0.111, avoiding quality degradation from sequential generation and fitting.
  • Cross-representation consistency. T2BM achieves 89.9/55.3 mm MPJPE/PA-MPJPE, substantially outperforming OmniMM cascades. Although T2M-GPT → OmniMM M2B achieves slightly lower MPJPE, its generation quality is worse.
  • Efficiency. T2BM takes 4.40 s per clip, compared with 1.06–3.23 s for learned cascades. While slower, it achieves better generation quality and directly generates both representations without a separate fitting stage.

Overall, synchronized generation provides a favorable balance of generation quality and cross-representation consistency, while directly producing both representations without a separate fitting stage.

Revision (Main paper). We have added the comparison with the learned cascades to the main paper (Sec. 6.2, Table 8).

AuthorsOpenReview ↗

R1: advantages of unified modeling over simpler alternatives (continue)

Architectural ablations. We appreciate the reviewer's suggestion regarding architectural ablations. OmniMM's primary contribution lies in unifying language, parametric motion, and biomechanical skeleton motion within a single framework, rather than introducing individual architectural components in isolation. While additional ablations could provide further insights into specific design choices, we believe they are complementary to validating our central contribution. Moreover, each complete multi-stage training run requires approximately 57 hours on four L40S GPUs (Appendix A.1.2), making extensive architectural ablations impractical within the rebuttal period.

Nevertheless, we ablate the two components most specific to our design, each replaced by a common alternative while keeping everything else fixed:

  • Motion tokenizer. CausalTAE substantially improves fitting accuracy over MLD-VAE [1], as adopted in MotionGPT3 [2], by enabling fine-grained, chunk-level motion tokenization and causal temporal modeling.
TokenizerMPJPE (m)↓PA-MPJPE (m)↓Accel. (m/s²)↓
MLD VAE0.1720.1097.615
CausalTAE (ours)0.0800.0545.280
  • Generation head. Our flow-matching head consistently outperforms the diffusion alternative across all T2B generation metrics.
HeadFID↓R@1↑R@3↑MMDist↓Diversity→
Diffusion0.1370.4320.6911.3992.400
Flow matching (ours)0.0630.5110.7641.2582.494

Revision (Appendix). We have added these ablations to Appendix E.1 (Tables 19 and 20).

References

[1] X. Chen et al. Executing Your Commands via Motion Diffusion in Latent Space. CVPR, 2023.

[2] B. Zhu et al. MotionGPT3: Human Motion as a Second Modality. arXiv:2506.24086, 2025.

AuthorsOpenReview ↗

R2: conversion tasks, metrics, and runtimes in Table 5

We thank the reviewer for the careful reading of Table 5. We clarify below that the optimization baselines and OmniMM address the same conversion tasks, with SKEL serving only as an intermediate step for the baselines.

Why an intermediate step is needed. There is no direct fitting method between BSM and SMPL. The only native SMPL-to-BSM route is AddBiomechanics marker fitting, which takes about 24 minutes per 10-second clip (Appendix B.5) and is therefore not practical for evaluation. We thus use SKEL as an intermediate step, because it allows us to derive an efficient and accurate optimization baseline.

SKEL connects BSM and SMPL rather than introducing a different representation. SKEL is a body model that combines the BSM skeleton with the SMPL mesh: it re-rigs the SMPL mesh with the BSM skeleton and is posed by the same 46 anatomical coordinates as BSM [1]. It is therefore tied to both representations:

  • SKEL and BSM. SKEL is parameterized by the same anatomical coordinates as BSM. Therefore, fitting SKEL directly estimates the biomechanical pose coordinates underlying the BSM representation, rather than an unrelated intermediate representation.
  • SKEL and SMPL. SKEL inherits SMPL's surface mesh topology and shape space. Therefore, SMPL-compatible joint positions can be obtained by applying the SKEL's SMPL joint regressor to the SKEL surface mesh, enabling direct comparison in the SMPL joint space.

Consequently, both methods perform the same conversion in each direction:

  • M2B. The baseline fits SKEL, and hence BSM coordinates, to the ground-truth SMPL joints. OmniMM maps SMPL to BSM, which is then mapped to SKEL joints for scoring, in the same way as the ground-truth BSM reference.
  • B2M. SKEL is the only available optimization-based bridge from BSM motion to SMPL joints, obtained by fitting SKEL and applying its mesh-based joint regressor. OmniMM instead directly predicts the full 77-D SMPL-H representation, which is decoded into SMPL joints using the same procedure as the ground-truth reference.

Positional metrics are the standard protocol. Comparing body representations by 3D joint position error is standard practice [1, 2]. In particular, SKEL [1] itself offers this comparison, evaluating joints obtained from SMPL against BSM joints in joint space. Joint space is also where both methods can be scored against the same reference.

Runtimes. Runtime comparisons include only the optimization step for the baselines and the generation step for OmniMM, excluding joint regression and other joint-space post-processing steps.

Full representation conversion. OmniMM predicts complete 57-D BSM or 77-D SMPL-H motion, rather than joint positions alone. Joint-space metrics are computed by decoding these full representations, providing a consistent measure of cross-representation fitting accuracy.

Therefore, Table 5 compares the same conversion tasks against the same reference under the standard protocol, demonstrating that OmniMM performs full representation conversion with joint-space accuracy comparable to optimization at a fraction of its fitting time.

Revision (Appendix). We have clarified the runtime accounting in Appendix B.3, explicitly distinguishing per-sequence optimization time from model generation time and specifying the excluded evaluation mappings.

References

[1] M. Keller et al. From Skin to Skeleton: Towards Biomechanically Accurate 3D Digital Humans. ACM TOG, 42(6), 2023.

[2] Y. Xia et al. Reconstructing Humans with a Biomechanically Accurate Skeleton. CVPR, 2025.

A1: comparison with a cascaded T2M + M2B pipeline

We thank the reviewer for this question. We have conducted this comparison, and the full results on generation quality, cross-representation consistency, and inference time are reported in R1.

A2: effect of prediction targets and procedures on Table 5

We thank the reviewer for this question. Both parts are addressed in R2, which explains how the prediction targets and procedures relate through SKEL, why the accuracy and runtime comparisons are fair, and why the results demonstrate full representation conversion assessed in joint space.

Reviewer_gSVdOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces OmniMM, a unified framework that jointly models anatomy-driven biomechanical skeleton motion, surface-driven parametric motion, and language. To enable this at scale, the authors construct paired parametric-biomechanical-text data by fitting the BSM anatomical skeleton model to SMPL surface markers via AddBiomechanics, exploiting the shared marker observation space of the two fitting paradigms. Architecturally, both motion streams are encoded as continuous tokens with causal temporal VAEs and processed by a MoT backbone with modality-specific towers. Experiments report competitive generation in both representations, strong understanding scores, and fitting accuracy comparable to optimization-based baselines at much faster inference.

Critical strengths

[Originality]

  • Treating biomechanical skeleton motion as a generative modality jointly modeled with parametric motion and language is an interesting idea to me. Prior work either injects biomechanical priors into generation or estimates skeletons from images, but none of them generate biomechanical motion itself or translate between the two representations with a learned model. I personally like this direction.

[Quality]

  • The experimental coverage is comprehensive: all proposed tasks are evaluated with retrained baselines, and the numbers are strong across the board. The data-quality validation is careful, and the anatomical gap analysis honestly quantifies why surface-driven statistics don't imply anatomical plausibility, which I found interesting.
Critical weaknesses

[Derived-modality circularity]

  • Regarding the entire manuscript, my main concern is that the "biomechanical modality" is not independently observed: it is produced by running AddBiomechanics on markers derived from the SMPL sequences themselves. The model is then evaluated against ground truth from that same pipeline, which natively explains why the model acehives comparable or better fitting accuracy as it essentially distills the optimizer. This is a minor part, yet an evaluation on biomechanical data from an independent source would make the claims in the paper more solid.
Questions for authors

I have no specific questions for authors. Overall I find this a work that solves an interesting problem with anatomically grounded motions. I thus lean towards weak accept.

6 comments
AuthorsOpenReview ↗

R1: Evaluation on independently observed biomechanical data

We thank the reviewer for the positive assessment and for this constructive suggestion. We acknowledge that our large-scale biomechanical training data is derived from SMPL. However, this reflects a broader challenge in the research community: to the best of our knowledge, no large-scale dataset currently provides language-annotated BSM motion paired with accurate parametric motion. Our shared-marker pipeline addresses this data gap, and we further validate its generalization through an independent-data evaluation below.

Evaluation on independent motion-capture data. Following the reviewer's suggestion, we additionally evaluate OmniMM on OpenCap [1], which provides real optical motion-capture markers independently recorded from human subjects.

  • Protocol. We select 10 motion sequences from 9 subjects, covering squats, sit-to-stand, and drop jumps. For each sequence, we independently fit SMPL-H using MoSh++ [2] and BSM using AddBiomechanics [3], both directly from the recorded markers. Neither representation is derived from the other, eliminating the SMPL-to-synthetic-marker dependency in our original data construction pipeline. We evaluate OmniMM without fine-tuning against the optimization baseline in both M2B and B2M, following the protocol of Table 5. OpenCap is entirely unseen during training and the results are shown in the table below.
  • Marker configuration. Since BSM does not provide an OpenCap-specific marker set, we construct one using anatomical correspondences. Lower-limb marker offsets follow the OpenCap musculoskeletal model, which shares the lower-limb segment frames with BSM. The C7, acromion, and wrist markers reuse the offsets of their corresponding BSM markers, while the sternum markers are positioned relative to C7 in the neutral pose, and the elbow markers are defined in a local coordinate frame constructed from the arm joint centers.
  • Data limitations. Like many existing biomechanical datasets, OpenCap primarily focuses on lower-body motion, resulting in missing or less reliable measurements of upper-body motion. Specifically, its marker set lacks markers on the head, hands, toes, and lumbar spine. Consequently, nine BSM coordinates remain unobserved and are fixed at their neutral values, while lumbar and scapular coordinates are weakly constrained. Both reference representations also contain residual fitting errors, introducing additional uncertainty into the evaluation.
TaskMethodMPJPE (m)↓PA-MPJPE (m)↓Accel. (m/s²)↓
M2BOpt.-based0.1220.0787.286
M2BOmniMM0.1380.0849.060
B2MOpt.-based0.1180.08511.71
B2MOmniMM0.1110.0825.780
  • Consistent performance relative to optimization. The results are consistent with Table 5, showing that OmniMM achieves comparable or better performance than optimization-based baselines. This further demonstrates OmniMM's ability to generalize to independently reconstructed motion pairs beyond our original SMPL-derived training pipeline. Absolute MPJPE increases by approximately 1.4–1.7× relative to Table 5 for both methods. This suggests that the increased difficulty is not specific to our model, potentially reflecting uncertainty in the independently fitted references and differences between OpenCap activities and our training distribution.

Although this evaluation is limited to 10 sequences, it provides evidence of generalization to an independent data source. Importantly, despite being trained on SMPL-derived BSM data, OmniMM generalizes to independently fitted SMPL-H and BSM representations from real-world observations, achieving accuracy comparable to or better than optimization-based methods. This suggests that OmniMM learns transferable relationships between motion representations beyond its training data construction pipeline. We acknowledge that developing large-scale biomechanical datasets from independent data sources, with rich annotations and corresponding parametric measurements, remains an important direction for the broader research community.

AuthorsOpenReview ↗

R1: Evaluation on independently observed biomechanical data (continue)

Revision (Main paper and Appendix).

We have included the OpenCap quantitative comparison with optimization-based baselines and the key findings in the main paper (Sec. 6.2, Table 9).

The evaluation protocol and implementation details, including marker-set construction, fitting procedures, and data limitations, are provided in Appendix B.4.

We also explicitly acknowledge the SMPL-derived nature of our large-scale biomechanical training data in the limitations (Sec. 7).

References

[1] S. D. Uhlrich et al. OpenCap: Human Movement Dynamics from Smartphone Videos. PLOS Computational Biology, 2023.

[2] N. Mahmood et al. AMASS: Archive of Motion Capture as Surface Shapes. ICCV, 2019.

[3] K. Werling et al. AddBiomechanics: Automating Model Scaling, Inverse Kinematics, and Inverse Dynamics from Human Motion Data through Sequential Optimization. PLOS ONE, 2023.

Dear Authors,

Thanks for your rebuttal! Your responses address my concerns and I'm glad to see that OmniMM shows the generalization to different data sources. I will keep my score and remain positive for this submission!

Comments on the paper

1 comment
AuthorsOpenReview ↗

Overall Response (to all reviewers)

We sincerely thank all reviewers for their time and constructive feedback. We are encouraged that the reviewers found treating biomechanical skeleton motion as a generative modality an interesting direction (gSVd), recognized that our paired dataset helps address the scarcity of paired biomechanical data (Edb5, Hx14), and found the experimental coverage comprehensive, with retrained baselines under controlled conditions (gSVd, Hx14).

Since the reviewers raised distinct questions, we respond to each reviewer individually, with new experiments and clarifications. The reviewers' comments have helped us improve the paper, and the revised paper incorporates them as follows:

Main paper

  • gSVd: Evaluation of both fitting directions on independently recorded OpenCap markers (Sec. 6.2, Table 9), and a statement in the limitations that our biomechanical data are derived from SMPL-based markers (Sec. 7).
  • Edb5: Comparison of synchronized generation with cascaded pipelines (Sec. 6.2, Table 8).
  • Hx14: Corrected Table 4, with one reference row per evaluator and the re-scored MotionGPT3 rows; the human-caption rows are no longer described as an upper bound.
  • Hx14: Standard deviations of R-precision over the evaluation replications in Tables 3 and 4.

Appendix

  • Edb5: Clarification of the runtime accounting in the fitting protocol (Appendix B.3).
  • gSVd: Protocol, marker configuration and data limitations of the OpenCap evaluation (Appendix B.4, Table 14).
  • Hx14: Dataset split construction and per-dataset composition (Appendix B.6, Table 17).
  • Edb5: Ablations of the motion tokenizer and the generation head (Appendix E.1, Tables 19 and 20).

We have also fixed typos in the paper.

We hope our responses address the reviewers' concerns, and we are happy to answer any further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.