acceptodds
Under review as a conference paper at ICLR 2027

One for All: Generalist Foundation Model for Cross-Sensor Skeleton Representation Learning

Abstract

For learning generalizable motion representations from large-scale unlabeled data, Self-supervised learning (SSL) has become a widely adopted methodology. However, existing approaches are primarily limited by the inherent heterogeneity of skeleton data—characterized by varying joint counts, indexing protocols, and topological structures across different sensors—which typically necessitates training separate, sensor-specific, or even entirely dataset-specific models. To overcome this, we introduce SOfA (Skeleton One for All), the first generalist foundation model designed to achieve sensor-unified skeleton representation learning across diverse sensors. To accommodate the dimensional gap caused by varying joint counts, we introduce a fixed-size set of learnable Canonical Joint Slots, acting as a universal vessel that seamlessly accommodates arbitrary skeletal topologies. SOfA fills these slots via an attention mechanism that dynamically aggregates skeletal information from sensor-specific inputs. Furthermore, we resolve joint index misalignment between various sensors by introducing a Semantic Joint Embedding derived from a pre-trained text encoder, rather than relying on absolute positional embeddings. To validate our approach, we standardized ten 3D skeleton datasets for unified training. Extensive experiments demonstrate that SOfA can serve as a truly universal encoder, achieving state-of-the-art (SOTA) performance across a wide range of downstream tasks and sensor types, often outperforming dataset-specific specialist models with a single foundation model.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Skeleton-Based Action Recognition, Self-Supervised Learning, Skeleton Foundation Models, Cross-Sensor Generalization

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_kF7qOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes SOfA, one self-supervised encoder for skeletons with different joint counts and orderings. Joint names are encoded by T5 as positional embeddings, and a fixed set of learnable canonical joint slots attends together with the input tokens, trained with DINO-style and masked slot distillation. It is pretrained on eight Kinect datasets with 20 or 25 joints and evaluated by linear probing, including two unseen 15-joint datasets.

Critical strengths

One frozen encoder transfers to skeleton layouts never seen in pretraining. With linear probing it reaches 91.7 on Florence and 97.0 on SBU, against 66.8/73.7 and 73.1/80.2 for SkeletonMAE and MAMP, and the ablation shows the semantic joint embedding matters for 20-joint data (NW-UCLA 72.3 → 92.9). The design is simple and practical for mixing datasets.

Critical weaknesses
  1. The comparison does not separate architecture from data. SOfA is pretrained on eight datasets, every baseline in Table 2 on one. Even so it trails AMR on NTU-60 X-Sub (87.0 vs. 87.4) and NTU-120 X-Sub (79.5 vs. 81.1). The clear gain is on PKU-MMD (67.1 vs. 60.3), which the paper attributes to data transfer. MAMP pretrained on the same pooled data is the missing control.

  2. The scope and impact are narrow. "Sensor-unified" covers only Kinect v1/v2 depth-camera skeletons with 15, 20 or 25 joints from lab datasets; LiDAR and wearable sensors appear in the motivation but not in the data, and there is no motion-capture, SMPL or video-estimated pose data. Linear-probe classification on these benchmarks has limited relevance for ICLR in the current era of large motion-language models.

Overall, I think the topic too narrow considering the current powerful MLLMs, including motion MLLMs, pose MLLMs, video MLLMs. This work is expected to be of very limited impact.

Questions for authors
  1. How does a sensor-specific method such as MAMP perform when pretrained on the same pooled 25-joint data, which makes up about 90% of the corpus?

  2. How were SkeletonMAE and MAMP applied to the 15-joint Florence and SBU data in Table 4? Given how small these datasets are, what is the variance over runs?

  3. Does the approach extend to non-Kinect skeletons, such as motion capture or 2D poses estimated from video?

3 comments
AuthorsOpenReview ↗

Response to Reviewer kF7q

We thank the reviewer for the helpful comments and specific questions. We have conducted additional experiments and provide the requested implementation details and results below, with full settings in the common responses. We will incorporate these results and clarifications into the revised manuscript.

W1 / Q1. MAMP pretrained on the pooled 25-joint data

We retrained MAMP on the complete 25-joint pool of SOfA's pretraining corpus. It achieves 81.7% on NTU-60 X-Sub and 71.6% on NTU-120 X-Sub. Please see Common response 2 for all five evaluation protocols, the pretraining-data scope, and encoder capacity and compute.

We agree that SOfA does not consistently exceed the strongest NTU specialists. We will acknowledge this and frame the contribution around a shared joint-stream encoder across topologies, with its broader transfer evaluated below and in Common response 1.

Q2. Baseline adaptation to 15-joint inputs and repeated-run variance

SkeletonMAE and MAMP use encoders pretrained on the NTU-120 X-Sub training split. For Florence and SBU-Inter, we map each of the 15 joints to the anatomically closest joint-name counterpart in the pretrained 25-joint layout and fill the remaining positions with zeros. The encoders remain frozen; only the downstream linear classifier is trained. We will clarify the pretraining setup and joint mapping in the revised manuscript.

The table reports mean ± standard deviation of Top-1 accuracy (%) over three downstream linear-probe training seeds (0, 1, 2). Pretrained checkpoints and evaluation splits are fixed across runs; only probe training varies.

EncoderFlorenceSBU-Inter
SkeletonMAE66.8 ± 3.573.1 ± 2.4
MAMP73.7 ± 3.180.2 ± 1.9
SOfA91.7 ± 0.897.0 ± 0.4

All three methods show greater variability on Florence, consistent with its smaller dataset size. We will include these repeated-run results in the revised manuscript.

W2 / Q3. Broader skeleton sources and downstream value

We thank the reviewer for this helpful comment. We agree that evaluation limited to action recognition on depth-sensor 3D skeletons provides limited evidence of broader applicability. Following the reviewer's suggestion, we conducted additional experiments on (1) motion generation with SMPL-based and MoCap skeletons and (2) 17-joint 2D skeletons estimated from video. Please see Common response 1 for the detailed settings and results.

(1) Motion generation with unseen SMPL-based and MoCap skeletons. HumanML3D uses an SMPL-based 22-joint layout, while KIT-ML uses a 21-joint MoCap representation. Both datasets and formats are absent from SOfA pretraining. We reuse the paper's SOfA/SOfA-L encoders without additional pretraining or fine-tuning.

These motion-generation datasets do not provide the action-class labels required by our recognition protocol. We therefore examine whether SOfA's features improve motion generation, extending evaluation from high-level action understanding to low-level generation. We compare JiT-style and PiT-style backbones trained with and without training-only REPA supervision from the frozen encoders. For each dataset and backbone, these runs use the official training/test splits and the same training schedule. Please see Common response 1(1) for the settings and results.

(2) Video-estimated 17-joint 2D skeletons. We evaluate NTU-60 X-Sub using HRNet-estimated 17-joint 2D skeletons, setting the third coordinate channel to z=0z=0 to fit SOfA's 3D input interface. HRNet-estimated 17-joint 2D observations are not used in SOfA pretraining. Frozen SOfA/SOfA-L encoders achieve 79.9%/82.3% linear-probe accuracy; SOfA and SOfA-L reach 92.3 ± 0.2% and 92.9 ± 0.1%, respectively, with full fine-tuning (seeds 0, 1, 2). We distinguish frozen-feature transfer from supervised adaptation; please see Common response 1(2) for details.

Thanks for the response, which addresses my main concerns. Although I still encourage the authors to conduct the study on multimodal action modeling (where the true challenges lie), I raise my rating for this paper, considering the current efforts the authors have made.

We sincerely thank the reviewer for the constructive feedback and for recognizing our efforts to address the concerns. We also appreciate the suggestion on multimodal action modeling and consider it an important direction for future work. We will incorporate the additional experimental results into the revised manuscript.

Reviewer_JCqbOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes SOfA, a self-supervised framework for learning unified skeleton representations across heterogeneous sensor configurations. It introduces Canonical Joint Slots (CJS) to aggregate skeleton inputs with different joint structures into fixed-size representations and Semantic Joint Embedding (SJE) to align joint semantics across sensors. A shared encoder is pretrained on multiple skeleton datasets and evaluated on both seen and unseen skeletal configurations.

Critical strengths
  1. The combination of CJS and SJE provides an interesting approach to handling heterogeneous skeleton structures within a unified representation learning framework.

  2. The paper integrates multiple skeleton datasets with different joint configurations into a shared pretraining framework, enabling representation transfer across datasets without retraining separate encoders.

  3. The experiments cover multiple datasets, different joint configurations, unseen skeleton protocols, and several downstream tasks, supported by systematic ablation studies.

Critical weaknesses
  1. SOfA is pretrained on a substantially larger and more diverse corpus than existing sensor-specific baselines. Comparisons under matched pretraining data and model capacity would help isolate the contributions of the proposed architecture.

  2. Although the paper evaluates two unseen 15-joint datasets and a synthetic 30-joint topology, these experiments cover a relatively limited range of sensor configurations. Further evaluation on more diverse real-world sensors would better support its universal applicability claims.

  3. Table 2 shows that SOfA does not consistently outperform recent sensor-specific methods on NTU-60 and NTU-120. Moreover, the modified NTU-60 X-View* protocol makes direct comparison with results under the original protocol less straightforward. These differences should be more clearly acknowledged.

Questions for authors

See Weaknesses

2 comments
AuthorsOpenReview ↗

Response to Reviewer JCqb

We thank the reviewer for the helpful comments on pretraining fairness, sensor coverage, and NTU evaluation. We have conducted additional experiments in response and will incorporate the results and clarifications into the revised manuscript. We summarize the relevant additions below and refer to the common responses for full details.

W1. Pretraining data and encoder capacity

We retrained MAMP on the complete 25-joint pool of SOfA's pretraining corpus. Please see Common response 2 for the five evaluation protocols, pretraining-data scope, and encoder capacity and compute. The encoder parameter counts are close (6.34M versus 6.56M), and the common response reports the corresponding accuracy and compute comparisons. We will clarify these settings in the revised manuscript.

W2. Sensor diversity and broader transfer

We thank the reviewer for this helpful suggestion. We agree that evaluation limited to depth-sensor 3D action recognition provides limited evidence of broader applicability. In response, we added motion generation with unseen SMPL-based and MoCap formats and action recognition with video-estimated 2D skeletons; please see Common response 1.

(1) From high-level understanding to low-level generation. We reuse the original frozen SOfA/SOfA-L encoders on HumanML3D (SMPL-based, 22 joints) and KIT-ML (MoCap, 21 joints), with no additional encoder pretraining or fine-tuning. Both datasets and formats are unseen during pretraining. Since these motion-generation datasets lack the action-class labels required by our recognition protocol, we test whether SOfA's features improve motion generation, comparing JiT-style and PiT-style generators with and without training-only REPA supervision. Both variants use the official train/test splits and the same training schedule for each dataset and backbone. Please see Common response 1(1) for the results.

(2) Video-estimated 17-joint 2D skeletons. We evaluate NTU-60 X-Sub using HRNet-estimated 17-joint 2D skeletons, setting the third coordinate channel to z=0z=0 to fit SOfA's 3D input interface. HRNet-estimated 17-joint 2D observations are not used in SOfA pretraining. Frozen SOfA/SOfA-L features achieve 79.9%/82.3% linear-probe accuracy; SOfA and SOfA-L reach 92.3 ± 0.2% and 92.9 ± 0.1%, respectively, with full fine-tuning (seeds 0, 1, 2). Please see Common response 1(2) for the settings and results.

W3. NTU results and the X-View* protocol

We agree that SOfA does not consistently outperform recent sensor-specific methods on NTU-60/120 X-Sub. For example, AMR reports 87.4/81.1, versus SOfA's 85.7/79.2. We will explicitly acknowledge this and emphasize the contribution of one shared encoder across sensors and topologies.

Why X-View* is used. Please see Appendix C and Fig. 5(f) for details. A sequence can be in the training split of one NTU protocol and the test split of another. Our shared NTU pretraining pool uses the intersection of the NTU-120 X-Sub and X-Set training splits, which contains 5,392 sequences from the official NTU-60 X-View test set. With this pretraining pool fixed, we exclude those sequences from evaluation, leaving 13,568 sequences, denoted X-View*. This filtering is completed before evaluation and does not depend on model predictions.

We acknowledge that X-View* changes the test set and limits direct comparison with published original X-View results. Common response 2 provides a comparison on the same X-View* test set: MAMP 86.0 versus SOfA 92.1. We will distinguish X-View and X-View* throughout the revised manuscript and bold results only within the same test protocol.

Reviewer_JCqbOpenReview ↗

Response to The Authors

Thank you for the detailed clarification and additional experiments. The pooled-data MAMP comparison strengthens the empirical support, while the evaluations on unseen skeleton formats, 2D inputs, and motion generation broaden the evidence for transferability. The clarification of X-View* also helps establish the scope of the comparisons. Overall, the response substantially addresses my concerns and reinforces the value of a shared encoder for cross-topology skeleton representation learning.

Reviewer_aH1VOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SOfA, a self-supervised foundation model that handles skeleton data with different sensors, joint counts, and joint indexing using a single encoder. It introduces (1) a fixed-size set of learnable Canonical Joint Slots (CJS), processed with the sensor tokens through attention so the output size is independent of the input skeleton, and (2) a Semantic Joint Embedding (SJE) that replaces index-based positional embeddings with T5 embeddings of joint names. Trained with a DINO-style objective on eight datasets, it is evaluated by linear probing on NTU, PKU-MMD, NW-UCLA, UT-Kinect, ETRI, and two held-out 15-joint datasets (Florence, SBU), plus semi-supervised, retrieval, and synthetic 30-joint experiments.

Critical strengths
  1. The problem is well motivated: existing skeleton SSL methods are tied to a fixed joint set and trained per dataset, so a single encoder across heterogeneous formats is a practical and under-explored goal.
  2. The method is simple and sensible. The SJE ablation shows a large effect on 20-joint data (NW-UCLA 92.9 vs. 72.3 without it), supporting the claim that index misalignment is a real obstacle.
  3. The results are strong and broad: one model is competitive with dataset-specific state of the art on NTU-60/120, clearly better on PKU-MMD, and performs very well on 20-joint and held-out 15-joint datasets without fine-tuning the encoder.
  4. The ablation is convincing: the variant without CJS and SJE, trained on the same corpus and recipe, is clearly worse than both the full model and the dataset-specific baselines, so the gains do not merely come from more data or a stronger recipe.
  5. The figures are also clear and well designed.
Critical weaknesses
  1. The contributions of CJS and its associated supervision could be better disentangled. While Table 5 shows substantial improvements from adding CJS, the paper does not clearly specify how class and slot tokens are aggregated for linear probing. It would be helpful to clarify whether the improvement partly reflects increased feature dimensionality.
  2. The SSL objective is an off-the-shelf combination whose skeleton-specific motivation is missing, and whose two losses are never ablated. L_Cano (masked reconstruction of canonical slots against teacher features at 90% masking) and L_DINO (global distillation with Sinkhorn–Knopp prototype assignment and KoLeo regularization) are structurally the iBOT recipe, which is standard in vision SSL (i.e., MAE + DINO). The paper gives no skeleton-specific investigation for why both objectives are needed. It does not identify failure modes of pure MAE (e.g., overfitting to low-level coordinates) or pure DINO (e.g., insufficient local structure) on skeleton data.
  3. Table 2 explicitly reports "the joint modality alone, without any multi-stream ensemble." Yet, the skeleton SSL community treats joint, bone, and motion as complementary representations, and several baselines cited by the paper itself (UmURL, CMD, HiCo) are precisely multi-modal or multi-stream unification methods. A foundation model that unifies sensors but not skeleton representations covers one third of the standard representation space, calling it "truly universal" is an overclaim.
  4. MotionBERT is a widely recognized human-motion pre-trained foundation model that unifies heterogeneous data sources and multiple tasks (3D pose estimation, skeleton action recognition, mesh recovery) with a single encoder, reaching strong performance with only 1–2 linear layers. It is absent from the Related Work, and from all experiments.
Questions for authors
  1. Please decompose the contribution of the proposed learning framework with two additional ablations: first, the two objectives L_Cano and L_DINO, individually (Sec. 3.3, Eq. 6) , Table 5 currently ablates only CJS/SJE, never the losses; second, SOFA pre-trained on NTU-60 alone vs. on the full eight-dataset corpus. Because Tables 2–4 compare against single-dataset pre-trained baselines, the gains attributable to the unified architecture vs. to corpus diversity are currently confounded.
  2. Does the unified representation extend to the bone and motion streams. The two other standard skeleton representations (Table 2 explicitly reports "the joint modality alone")? If not, please scope the "generalist / truly universal encoder" claims to joint-only cross-sensor unification.
  3. Please provide a concrete scenario in which sensor heterogeneity cannot be resolved at the pose-estimation / data-standardization layer. Noting that MotionBERT already generalizes across datasets by standardizing inputs to a fixed skeleton format, without any architectural sensor unification. Second, a comparison of SOFA vs. MotionBERT as skeleton foundation models, ideally with numbers under a comparable protocol, or an explicit analysis of why the protocols are not directly comparable.
6 comments
AuthorsOpenReview ↗

Response to Reviewer aH1V (1/2)

We thank the reviewer for the constructive assessment and for recognizing the motivation, cross-topology results, and controlled architectural ablations. We clarify the readout, add the requested ablations, and summarize the relevant additions from our common responses below.

W1. Readout and feature dimensionality

The linear probe concatenates the class token with the mean of the canonical-slot tokens:

h=[wCLS;MeanPool⁡(Wc)]∈R2d.\mathbf{h}=\left[\mathbf{w}_{\mathrm{CLS}};\operatorname{MeanPool}(\mathbf{W}_c)\right]\in\mathbb{R}^{2d}.

Here, Wc∈RNc×d\mathbf{W}_c\in\mathbb{R}^{N_c\times d} contains all canonical tokens, with Nc=(T/PT)JcN_c=(T/P_T)J_c as defined in the manuscript. These tokens are mean-pooled, never flattened, so the probe dimension is independent of the input joint count and JcJ_c. The readout ablation on NTU-60 X-Sub is:

ModelCLS only (dd)Mean slot only (dd)CLS + mean slot (2d2d)
SOfA83.772.885.7
SOfA-L84.981.387.0

Adding pooled slots improves accuracy over CLS alone by 2.0/2.1 percentage points, consistent with complementary information. We will report the dimensions explicitly and distinguish these readout gains from the CJS/SJE architectural ablation in Table 5.

W2 / Q1(a). The two objectives

In the revised manuscript, we will cite iBOT [1], state that our objectives are inspired by iBOT, and explain the distinction between its patch-token targets and our shared canonical-slot targets. iBOT predicts teacher targets at corresponding masked input-patch positions; for skeleton tokens, those positions depend on the sensor's joint layout. In contrast, LCano\mathcal{L}_{\mathrm{Cano}} predicts teacher canonical-slot outputs whose learned identities are shared across topologies.

DINO loss aligns sequence-level representations across views, while Cano loss trains sparse observations to predict the shared slot representation. The requested loss ablation reports linear-probe accuracy (%):

ObjectiveNTU-60 X-SubNTU-120 X-SubPKU X-SubNW-UCLASBU-Inter
LDINO\mathcal{L}_{\mathrm{DINO}} only80.971.555.789.487.8
LCano\mathcal{L}_{\mathrm{Cano}} only62.450.745.185.679.4
Both (SOfA)85.779.265.492.997.0

Global distillation is stronger alone, but combining the objectives improves every benchmark. Adding canonical-slot prediction gives +4.8/+7.7/+9.7 points on NTU-60/NTU-120/PKU. This supports complementary supervision, without attributing the gains to an untested failure mechanism of coordinate-reconstruction MAE.

Q1(b). NTU-60-only versus full-corpus pretraining

We hold the architecture, objectives, readout, training recipe, and optimizer iterations fixed, changing only the pretraining corpus.

Pretraining corpusNTU-60 X-SubNTU-120 X-SubPKU X-SubNW-UCLASBU-Inter
NTU-60 only84.574.555.291.288.9
Full eight-dataset corpus85.779.265.492.997.0

The full corpus improves all five benchmarks at the same optimization budget. This measures the combined effect of data scale and diversity with the architecture fixed; Table 5 separately tests CJS/SJE under the same full corpus and recipe. SOfA's fixed-slot readout is designed for sensor-unified learning, with a different representation trade-off from retaining the full patch-token sequence.

Please see Common response 2 for MAMP retrained on SOfA's 25-joint pretraining pool with a similar encoder parameter count. SOfA achieves higher accuracy on all five reported protocols with lower encoder compute. The common response also explains how leakage prevention reduces NTU-specific training data despite enlarging the overall corpus.

W3 / Q2. Joint-stream scope

Bone and motion streams are derived from joint coordinates: bone vectors use differences between connected joints, and motion uses temporal differences. They provide complementary cues from the same observations. Our current scope is joint-stream, cross-topology unification. We thank the reviewer for the suggestion and will identify extending the framework to these complementary streams as future work. We will replace “truly universal” with “a shared encoder for joint-stream, cross-topology representation learning.”

AuthorsOpenReview ↗

Response to Reviewer aH1V (2/2)

W4 / Q3. MotionBERT and input standardization

We will cite and discuss MotionBERT [2] in the Related Work section of the revised manuscript. The table summarizes its input-format guidance and our new 2D results; please see Common response 1(2) for the discussion.

The 2D evaluation uses NTU-60 X-Sub with HRNet-estimated 17-joint skeletons; SOfA takes (x,y,0)(x,y,0) inputs.

AspectMotionBERTSOfA / SOfA-L
Input joint formatFixed 17-joint H36M format; other formats require conversionVariable joint sets with joint-name embeddings
Pretraining observationsNoisy, partial 2D keypoints used to recover 3D motion3D skeleton datasets; no HRNet 2D pretraining data
2D linear probe (frozen encoder)Not compared here79.9 / 82.3 (SOfA / SOfA-L)
2D full fine-tuning93.092.3 ± 0.2 / 92.9 ± 0.1 (SOfA / SOfA-L; seeds 0, 1, 2)

MotionBERT's pretraining directly covers 2D inputs. SOfA's frozen-encoder results demonstrate transfer to video-estimated 2D observations, while full fine-tuning evaluates downstream adaptation. We distinguish these protocols and make no claim of numerical superiority over MotionBERT. Common response 1(1) further reports frozen transfer to unseen 22-/21-joint MoCap formats and useful training-only supervision for motion generation.

A concrete scenario is pooling archived 3D skeleton recordings from 20- and 25-joint sensors without the original videos. MotionBERT's standard interface uses a fixed 17-joint 2D representation [3]. Converting these 3D observations to that interface requires joint remapping and projection: remapping cannot recover unmeasured landmarks, while projection discards depth information. When videos are available, estimating 2D keypoints from them introduces additional pose-estimation uncertainty. SOfA directly encodes the available 3D joint set, providing one shared encoder for heterogeneous observations without an additional 2D pose-estimation or projection stage.

References

[1] Jinghao Zhou et al. iBOT: Image BERT Pre-Training with Online Tokenizer. ICLR 2022.

[2] Wentao Zhu et al. MotionBERT: A Unified Perspective on Learning Human Motion Representations. ICCV 2023.

Thank you for the rebuttal. After revisiting the manuscript, I still have two questions on technical novelty and the advantage of this joint-based foundation model, no additional experiments required.

  1. What does SOFA retain from the iBOT dual-objective recipe (masked-token distillation + sequence-level DINO distillation), and what is specific to skeleton representation? Please explain conceptually why the shared canonical-slot target L_Cano is needed where iBOT's position-aligned patch target does not transfer across sensor joint layouts, and state the paper's actual novelty claim once the slot design is separated from the general SSL recipe.

  2. Advantage over MotionBERT-type models. Within the joint-only scope, what is SOfA's concrete advantage over models that standardize inputs to a fixed skeleton format and remap or convert other topologies? Please explain the practical gain of a cross-topology shared encoder. Note that the two methods unify different axes: MotionBERT unifies data sources and tasks on a fixed topology, whereas SOFA unifies topology within a single joint stream.

I appreciate the authors' efforts and look forward to further clarification.

Comments on the paper

5 comments
AuthorsOpenReview ↗

Common Response to All Reviewers

We thank the reviewers for their constructive feedback and for recognizing the value of a shared encoder across heterogeneous skeleton observations. We apologize for the delayed response, as completing the additional experiments across multiple datasets, input formats, and downstream tasks required more time.

We address the common concerns about pretraining data and model capacity, generalization to new skeleton formats and 2D observations, and downstream utility below. Reviewer-specific questions and ablations are addressed in the individual responses.

We will incorporate the additional experimental results, analyses, and clarifications into the revised manuscript and plan to upload a consolidated revision before the discussion period ends.

AuthorsOpenReview ↗

Common Response to All Reviewers (1/3)

C1. Broader skeleton formats and motion generation (1/2)

We agree that broader input formats and downstream tasks are needed to assess SOfA's utility beyond depth-sensor 3D action recognition. In response, we add (1) motion generation with unseen SMPL-based 22-joint and MoCap 21-joint formats and (2) action recognition with 17-joint 2D skeletons estimated from video. The first extension moves from high-level understanding to low-level generation using the original frozen SOfA/SOfA-L encoders, testing their usefulness across new data sources, joint layouts, and tasks.

(1) Unseen motion-capture skeletons and transfer to generation

Setting. We evaluate HumanML3D [1] (SMPL-based, 22 joints) and KIT-ML [2] (MoCap, 21 joints), both absent from SOfA pretraining. We reuse the paper's SOfA/SOfA-L encoders without additional pretraining or fine-tuning and retain each dataset's native joint layout. These motion-generation datasets lack the action-class labels required by our recognition protocol. We therefore evaluate whether SOfA's features improve motion generation, using unconditional noise-to-motion generation to assess their value beyond action understanding.

We adapt JiT [4] and PiT from PixelDiT [5] into a JiT-style motion backbone and a PiT-style motion backbone. Each generator is trained on its dataset's official training split and evaluated on the official test split. For each dataset and backbone, we use the same training schedule with and without REPA [3] supervision from SOfA/SOfA-L. REPA adds a loss that aligns the generator's intermediate features with those extracted from clean motion by frozen SOfA. This tests whether SOfA's representations improve generation on unseen skeletons. SOfA provides training-only supervision and adds no inference cost.

Evaluation. We report mean ± std over three seeds (0, 1, 2). All metrics are lower-is-better; real-motion rows give kinematic references. Bold marks the best generated results per backbone.

  • FID: Fréchet distance between generated and ground-truth motion feature distributions, using the widely used pretrained motion evaluator of Guo et al. [1], independent of SOfA.
  • Skate: Unitless fraction of frames where a contacting foot joint (height <5 cm) slides faster than 0.5 m/s.
  • Jerk: Mean norm of the third temporal difference of joint positions, averaged over joints and valid time indices and divided by the cubed frame interval (m/s3\mathrm{m/s^3}).

HumanML3D (SMPL, 22 joints).

REPA teacherJiT-style FID ↓JiT-style Skate ↓JiT-style Jerk ↓PiT-style FID ↓PiT-style Skate ↓PiT-style Jerk ↓
Real motion—0.10543.1—0.10543.1
None (base)0.644 ± 0.0440.167 ± 0.001134.5 ± 1.40.338 ± 0.0170.114 ± 0.000119.5 ± 0.1
SOfA0.356 ± 0.0190.136 ± 0.00199.8 ± 0.40.377 ± 0.1480.108 ± 0.00192.5 ± 6.9
SOfA-L0.399 ± 0.0340.136 ± 0.00699.3 ± 0.70.263 ± 0.0170.110 ± 0.00097.8 ± 2.4

KIT-ML (MoCap, 21 joints).

REPA teacherJiT-style FID ↓JiT-style Skate ↓JiT-style Jerk ↓PiT-style FID ↓PiT-style Skate ↓PiT-style Jerk ↓
Real motion—0.24458.4—0.24458.4
None (base)1.375 ± 0.4130.272 ± 0.00775.8 ± 1.61.773 ± 0.3680.261 ± 0.00081.3 ± 2.5
SOfA1.337 ± 0.0630.262 ± 0.00270.7 ± 1.01.112 ± 0.1360.247 ± 0.00174.2 ± 2.5
SOfA-L1.070 ± 0.1750.260 ± 0.00671.4 ± 1.41.139 ± 0.0470.252 ± 0.00371.6 ± 3.5

Results. Across both unseen layouts and both backbones, mean Skate and Jerk improve in all eight teacher–backbone–dataset combinations, and mean FID improves in seven of eight. FID drops by 44.7% on HumanML3D (JiT-style + SOfA) and 37.3% on KIT-ML (PiT-style + SOfA). The consistent kinematic gains show that frozen SOfA features support foot-motion consistency and temporal smoothness on unseen skeletons, extending their downstream value from action recognition to generation.

AuthorsOpenReview ↗

Common Response to All Reviewers (2/3)

C1. Broader skeleton formats and motion generation (2/2)

(2) Extension to video-estimated 2D skeletons (17 joints)

Setting. We evaluate NTU-60 X-Sub using HRNet-estimated 17-joint 2D skeletons from video.

Pretraining-domain difference. MotionBERT [6] is pretrained to reconstruct 3D motion from noisy, partial 2D keypoints, giving it direct experience with 2D inputs. SOfA is pretrained on 3D skeleton datasets. We represent each 2D joint as (x,y,0)(x,y,0) by setting the third (zz) channel to zero. HRNet-estimated 17-joint 2D observations are not included in SOfA's pretraining data. We evaluate both frozen-encoder linear probing and full fine-tuning to distinguish representation transfer from supervised adaptation.

Full fine-tuning results are mean ± std over seeds (0, 1, 2).

ModelDownstream evaluationAccuracy (%) ↑
MotionBERT [6]Full fine-tuning93.0
SOfALinear probe (frozen encoder)79.9
SOfA-LLinear probe (frozen encoder)82.3
SOfAFull fine-tuning92.3 ± 0.2
SOfA-LFull fine-tuning92.9 ± 0.1

Transfer beyond 3D inputs. The frozen SOfA/SOfA-L encoders achieve 79.9%/82.3%, showing that their learned features remain useful for 2D action recognition without encoder adaptation. With full fine-tuning, SOfA and SOfA-L reach 92.3 ± 0.2% and 92.9 ± 0.1%, respectively. SOfA-L's mean is 0.1 percentage point below MotionBERT's reported 93.0%, despite pretraining only on 3D skeleton datasets. These results support both transfer to video-estimated 2D observations and effective downstream adaptation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.