- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes SOfA, one self-supervised encoder for skeletons with different joint counts and orderings. Joint names are encoded by T5 as positional embeddings, and a fixed set of learnable canonical joint slots attends together with the input tokens, trained with DINO-style and masked slot distillation. It is pretrained on eight Kinect datasets with 20 or 25 joints and evaluated by linear probing, including two unseen 15-joint datasets.
Critical strengths
One frozen encoder transfers to skeleton layouts never seen in pretraining. With linear probing it reaches 91.7 on Florence and 97.0 on SBU, against 66.8/73.7 and 73.1/80.2 for SkeletonMAE and MAMP, and the ablation shows the semantic joint embedding matters for 20-joint data (NW-UCLA 72.3 → 92.9). The design is simple and practical for mixing datasets.
Critical weaknesses
-
The comparison does not separate architecture from data. SOfA is pretrained on eight datasets, every baseline in Table 2 on one. Even so it trails AMR on NTU-60 X-Sub (87.0 vs. 87.4) and NTU-120 X-Sub (79.5 vs. 81.1). The clear gain is on PKU-MMD (67.1 vs. 60.3), which the paper attributes to data transfer. MAMP pretrained on the same pooled data is the missing control.
-
The scope and impact are narrow. "Sensor-unified" covers only Kinect v1/v2 depth-camera skeletons with 15, 20 or 25 joints from lab datasets; LiDAR and wearable sensors appear in the motivation but not in the data, and there is no motion-capture, SMPL or video-estimated pose data. Linear-probe classification on these benchmarks has limited relevance for ICLR in the current era of large motion-language models.
Overall, I think the topic too narrow considering the current powerful MLLMs, including motion MLLMs, pose MLLMs, video MLLMs. This work is expected to be of very limited impact.
Questions for authors
-
How does a sensor-specific method such as MAMP perform when pretrained on the same pooled 25-joint data, which makes up about 90% of the corpus?
-
How were SkeletonMAE and MAMP applied to the 15-joint Florence and SBU data in Table 4? Given how small these datasets are, what is the variance over runs?
-
Does the approach extend to non-Kinect skeletons, such as motion capture or 2D poses estimated from video?
3 comments
Thanks for the response, which addresses my main concerns. Although I still encourage the authors to conduct the study on multimodal action modeling (where the true challenges lie), I raise my rating for this paper, considering the current efforts the authors have made.
We sincerely thank the reviewer for the constructive feedback and for recognizing our efforts to address the concerns. We also appreciate the suggestion on multimodal action modeling and consider it an important direction for future work. We will incorporate the additional experimental results into the revised manuscript.
Response to Reviewer kF7q
We thank the reviewer for the helpful comments and specific questions. We have conducted additional experiments and provide the requested implementation details and results below, with full settings in the common responses. We will incorporate these results and clarifications into the revised manuscript.
W1 / Q1. MAMP pretrained on the pooled 25-joint data
We retrained MAMP on the complete 25-joint pool of SOfA's pretraining corpus. It achieves 81.7% on NTU-60 X-Sub and 71.6% on NTU-120 X-Sub. Please see Common response 2 for all five evaluation protocols, the pretraining-data scope, and encoder capacity and compute.
We agree that SOfA does not consistently exceed the strongest NTU specialists. We will acknowledge this and frame the contribution around a shared joint-stream encoder across topologies, with its broader transfer evaluated below and in Common response 1.
Q2. Baseline adaptation to 15-joint inputs and repeated-run variance
SkeletonMAE and MAMP use encoders pretrained on the NTU-120 X-Sub training split. For Florence and SBU-Inter, we map each of the 15 joints to the anatomically closest joint-name counterpart in the pretrained 25-joint layout and fill the remaining positions with zeros. The encoders remain frozen; only the downstream linear classifier is trained. We will clarify the pretraining setup and joint mapping in the revised manuscript.
The table reports mean ± standard deviation of Top-1 accuracy (%) over three downstream linear-probe training seeds (0, 1, 2). Pretrained checkpoints and evaluation splits are fixed across runs; only probe training varies.
All three methods show greater variability on Florence, consistent with its smaller dataset size. We will include these repeated-run results in the revised manuscript.
W2 / Q3. Broader skeleton sources and downstream value
We thank the reviewer for this helpful comment. We agree that evaluation limited to action recognition on depth-sensor 3D skeletons provides limited evidence of broader applicability. Following the reviewer's suggestion, we conducted additional experiments on (1) motion generation with SMPL-based and MoCap skeletons and (2) 17-joint 2D skeletons estimated from video. Please see Common response 1 for the detailed settings and results.
(1) Motion generation with unseen SMPL-based and MoCap skeletons. HumanML3D uses an SMPL-based 22-joint layout, while KIT-ML uses a 21-joint MoCap representation. Both datasets and formats are absent from SOfA pretraining. We reuse the paper's SOfA/SOfA-L encoders without additional pretraining or fine-tuning.
These motion-generation datasets do not provide the action-class labels required by our recognition protocol. We therefore examine whether SOfA's features improve motion generation, extending evaluation from high-level action understanding to low-level generation. We compare JiT-style and PiT-style backbones trained with and without training-only REPA supervision from the frozen encoders. For each dataset and backbone, these runs use the official training/test splits and the same training schedule. Please see Common response 1(1) for the settings and results.
(2) Video-estimated 17-joint 2D skeletons. We evaluate NTU-60 X-Sub using HRNet-estimated 17-joint 2D skeletons, setting the third coordinate channel to z=0 to fit SOfA's 3D input interface. HRNet-estimated 17-joint 2D observations are not used in SOfA pretraining. Frozen SOfA/SOfA-L encoders achieve 79.9%/82.3% linear-probe accuracy; SOfA and SOfA-L reach 92.3 ± 0.2% and 92.9 ± 0.1%, respectively, with full fine-tuning (seeds 0, 1, 2). We distinguish frozen-feature transfer from supervised adaptation; please see Common response 1(2) for details.