- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper introduces OmniMM, a single model designed for handling language, SMPL-H motion, and biomechanical skeleton motion (BSM). This is achieved using a three-tower Mixture-of-Transformers that combines discrete text tokens with continuous tokens from two motion tokenizers, enabling text-conditioned generation, motion captioning, bidirectional conversion, and the simultaneous generation of both motion representations. The paired dataset is created by fitting BSM to synthetic markers placed on SMPL-H sequences from six datasets. The experiments compare generation and captioning performance with retrained baselines, assess fitting accuracy and speed, and investigate anatomical-limit violations in generated motion.
Critical strengths
- The unified model, together with the paired dataset, enables language-conditioned motion modeling to be applied to biomechanical skeletons. OmniMM supports generation, understanding, and conversion between motion representations using 26,253 paired clips totaling 51 hours (Table 1). The generated BSM retains the anatomical benefits of the target representation: for the evaluated elbows and knees, violations exceeding 10 degrees occur in 0.01–0.53% of generated BSM frames, compared with 21.4–55.1% for the generated parametric stream (Table 6).
- The experiments indicate useful improvements when compared against baseline models under controlled conditions. The learned baselines are retrained using their official implementations on the same representations and data splits (Appendix B.1). OmniMM achieves lower T2M/T2B FID than the compared baselines, although it does not lead on every retrieval metric (Table 3). Learned fitting takes 0.440/1.700 seconds for M2B/B2M, respectively, compared with 7.501/6.966 seconds for the 20-iteration optimization baseline (Table 5). B2M also improves accuracy, while M2B trades some accuracy for speed. The paper clearly states that joint generation does not impose the BSM stream’s anatomical constraints on the parametric stream.
Critical weaknesses
- The evaluation reference and scoring procedure in Table 4 require clarification. Although M2T and B2T have separate modality-specific evaluators, they share the same GT row. Its MMDist of 0.966 matches neither GT value in Table 3, which reports 0.943 for parametric motion and 1.151 for biomechanical motion. If the same ground-truth pairs and embeddings are used, reversing the retrieval direction alone would not explain this difference in mean distance. MotionGPT3 also has unusually high MMDist values of 2.634/2.424 despite strong retrieval scores. These observations raise questions about the reference samples, embedding normalization, and scoring pipeline, rather than establishing that the results are incorrect.
- Missing split details make the reported captioning gains difficult to assess. FineDance sequences are divided into windows with up to 50% overlap, and all subsequences of a dance receive the same two template captions. TRUMANS also uses template-based caption augmentation (Appendix B.4). The paper does not state whether subsequences from each original recording are kept within a single split or specify the test-set composition for Tables 3–4. Without this information, it is unclear whether closely related motion segments occur across splits or how strongly the captioning scores depend on repeated templates. R-precision is averaged over 20 evaluation replications, but no spread is reported, and variability across training runs is also unspecified.
Questions for authors
- Which evaluator and reference samples produced the shared GT row in Table 4, and why does it differ from Table 3? Please specify the embedding normalization and distance measure used for MMDist and Diversity, and confirm that all methods used the same scoring and post-processing pipeline. The human-caption reference should also be distinguished from a strict upper bound on retrieval performance.
- How were the training, validation, and test sets constructed for each source dataset? Were overlapping FineDance windows and other subsequences from the same original recording kept within the same split? Please provide the dataset composition and sample counts underlying Tables 3–4.
- Generated BSM has fewer anatomical-limit violations than ground truth for three of the four evaluated joints. Could the author compare their range-of-motion distributions to check whether the tokenizer’s joint-limit loss suppresses valid motion near the limits? Please also specify which clips or generated samples were used for Tables 6–7.
- Section 4 states that data construction follows the same surface-marker and fitting strategy as BioAMASS, but Table 2a reports lower marker errors, such as 1.08 versus 1.70 cm for CMU bony markers. Does this difference come from AddBiomechanics settings or versions, sequence filtering, or the error computation?
7 comments
R2: dataset splits and evaluation composition
We thank the reviewer for raising this concern. We address the dataset splits, the evaluation composition of Tables 3 and 4, the template captions, and the variability in turn, which also answers Q2.
Split construction and composition. For FineDance and TRUMANS, we split into training, validation and test sets at the level of the original recordings, before temporal segmentation. All windows derived from one recording therefore fall in the same split, and we verified that no recording or overlapping window is shared across splits. FineDance has 162 / 20 / 21 training / validation / test dances, and TRUMANS has 261 / 33 / 33 recordings. For AMASS, we use the official HumanML3D split without modification, following the standard protocol of prior text-to-motion and motion-to-text work. The remaining datasets use our own 8:1:1 training / validation / test splits. After the split, we filter out clips outside the valid length of 20 to 200 frames (1 to 10 s at 20 fps), which is the HumanML3D length filter. The table gives the number of test clips scored in Tables 3 and 4, and the number of clips used for training.
| Dataset | Training clips | Evaluated clips (Tables 3–4) |
|---|---|---|
| AMASS | 8,433 | 1,563 |
| CombatMotion | 6,192 | 703 |
| 100STYLE | 1,650 | 206 |
| FineDance | 2,776 | 275 |
| TRUMANS | 1,490 | 188 |
| HumanSC3D | 550 | 60 |
| Total | 21,091 | 2,995 |
Template captions. FineDance uses two captions per dance style, and TRUMANS uses template-based augmentation with motion-specific descriptions. Although caption templates may be shared between training and test sets, the FineDance and TRUMANS test recordings are entirely unseen during training, and together they make up only a small portion of the evaluated clips.
Evaluation variability. Over the 20 evaluation replications, the standard deviation of R-precision lies between 0.002 and 0.008 for every method and experiment.
Training variability. Each complete multi-stage training run takes about 57 hours on four L40S GPUs (Appendix A.1.2), which makes multiple independent runs impractical within the rebuttal period. We acknowledge this limitation.
Revision (Main paper and Appendix). We have added these splitting procedures and dataset statistics to Appendix B.6 (Table 17), and the standard deviations to Tables 3 and 4.
Answers to Q1 and Q2
A1: evaluator reference and scoring details
We thank the reviewer for this question. All parts are addressed in R1, which identifies the reference samples behind each GT row, specifies the embedding normalization and distance used for MMDist and Diversity, confirms the shared scoring pipeline, and distinguishes the human-caption reference from an upper bound.
A2: split construction and dataset composition
We thank the reviewer for this question. All parts are addressed in R2, which describes how each source dataset is split, confirms that all windows of a FineDance or TRUMANS recording stay within one split, and gives the per-dataset composition and sample counts underlying Tables 3 and 4.
R1: reference rows and scoring pipeline of Table 4
We thank the reviewer for the careful reading, which helped us identify two issues in Table 4. We address them below together with the questions in Q1. Neither issue changes the conclusions of the paper.
Scoring pipeline. All methods in Tables 3 and 4 are scored with the same test clips, the same 32-way R-precision over 20 replications, and the same post-processing. All methods also use the same frozen evaluators, except the MotionGPT3 rows of Table 4, which by our mistake were scored with an older version of our evaluator. We apologize for this error and have re-scored MotionGPT3 with the correct evaluator (see below), so that all methods are now evaluated identically.
Embedding normalization and distance. Each evaluator encodes a clip and a caption into 512-D embeddings, L2-normalizes both, and multiplies them by the evaluator's learned scale. The embedding-based evaluation metrics are computed using Euclidean distances:
Since the two evaluators have different scales, MMDist and Diversity should be compared across methods within one evaluator, not across the motion and biomechanics columns.
Clarification of GT reference scores. We thank the reviewer for spotting this. We reported only the parametric-motion GT scores in Table 4. Checking our evaluation process, we noticed a minor inconsistency in how the reference captions were chosen, which produced the slightly different GT numbers. Most test clips have several human captions, while the runs behind Table 3 use the first caption, whereas the run behind Table 4 draws one at random. The evaluator itself is deterministic, and re-scoring with the first caption reproduces Table 3:
The two reference MMDist values differ by only 0.023, a small gap compared with the differences between methods. This inconsistency affects only the printed reference rows and does not change the comparison between methods. The reviewer is also right that B2T needs its own reference, for which we now use the biomechanical GT scores already reported in Table 3.
Why MotionGPT3 has a high MMDist despite strong retrieval. We thank the reviewer for this observation. As noted above, the high MMDist comes from our mistake of scoring the MotionGPT3 rows with an older version of our evaluator. Re-scoring them with the correct evaluator gives:
With the corrected scores, MotionGPT3's MMDist is consistent with its retrieval accuracy, while OmniMM continues to outperform all baselines across all metrics on both tasks. Thus, the conclusions of Table 4 remain unchanged. We also correct the inconsistent decimal formatting in the MotionGPT3 rows.
Revision (Main paper). We now report one reference row per evaluator on a fixed caption per clip, and include the corrected MotionGPT3 rows (Table 4).
Revision (Appendix). We describe the human-caption rows as reference values in Appendix B.2.