- Rating
- 2(−1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper builds JohanssonBench, which renders NTU-60 and UTD-MHAD videos as point-light keypoint (JKP) stimuli, and shows that frontier MLLMs reach roughly half of a human baseline on them. It proposes RBI: a LoRA-tuned LTX-2 (19B) generates an RGB video from the JKP input, and a frozen MLLM then classifies the action. RBI is compared with LoRA fine-tuning of MLLMs on cross-instance, cross-action and cross-dataset splits.
Critical strengths
-
The benchmark is well designed: three splits with increasing distribution shift, an AMT human baseline (81%, Fleiss κ = 0.6), and a broad sweep of open and closed MLLMs. Showing that frontier models are near half of human level on point-light motion is a useful probe of how much MLLM action recognition relies on appearance.
-
Analysis-by-synthesis generalizes better than fine-tuning. On unseen actions (NTU-CA), RBI averages 80.8 versus 47.6 for LoRA-SFT across four MLLMs, close to the RGB upper bound of 88.3; on UTD it reaches 24.8 versus 13.8. Off-the-shelf results on event-camera data and gestures support the idea beyond one setting.
Critical weaknesses
-
The most natural baselines are missing. JKP is rendered from DWPose keypoints, so skeleton recognizers trained on the same 2D keypoints (e.g., ST-GCN or PoseC3D), and zero-shot skeleton methods for the cross-action split, are the obvious comparisons; only MLLM LoRA-SFT is used. No inference cost is reported either, although every test video requires a 19B video-generation pass.
-
Some claims go beyond the evidence. "Matches the human baseline" holds only on NTU-CI, where the generator is trained on 14,063 in-distribution pairs and SFT reaches the same 80.4 with InternVL3-8B; there is no human baseline on NTU-CA or UTD. On UTD, absolute accuracy stays low (20.9–32.4), and several emergent abilities are shown only qualitatively.
I am not very clear why we need this representation for analysis. It's very rare and unconvincing for me.
Questions for authors
-
How does a skeleton recognizer trained on the same DWPose 2D keypoints perform on all three splits, and how does a zero-shot skeleton method perform on NTU-CA?
-
What is the inference time and GPU cost per video for RBI compared with SFT?
-
Which NTU protocol defines the train/test split? Do NTU-CA test videos share actors and scenes with the generator's training data?
4 comments
Addressing the remaining questions
Here, we address the remaining questions.
I am not very clear why we need this representation for analysis. It's very rare and unconvincing for me.
We are happy to clarify the motivation and broader significance of our study on Johansson's keypoint representation. Our work is motivated by a fundamental question: do modern video models perceive biological motion from Johansson's sparse moving keypoints, as humans do? To measure this, we use Johansson's keypoints as testbed for probing pure temporal perception with no appearance information.
Broader significance. First, JohanssonBench tests for motion perception from sparse keypoints alone which can be indicate of real-world scenarios when appearance cues are unreliable, e.g., under low lighting, occlusion, or distant observations [1]. Second, beyond Johansson, RBI demonstrates the broader potential of generative reconstruction as an interpretable bridge between sparse motion and pretrained MLLMs. There are several benefits to the intermediate RGB generation:
- Generalization to novel tasks: Without any training, RBI is able to solve the psychophysical tests like the Hidden Bird Illusion (Fig. 5) or even work with novel stimuli like Dynamic-Vision Sensors (DVS).
- Interpretability: RGB output is human-interpretable rather than opaque latent features. As shown in Fig. 6 (newly added in the paper), we show that RBI, while generating RGB outputs, learns to map local keypoints accurately to human joints.
- Provides general-purpose visual interface to pretrained MLLMs: Generating RGB videos allows us to use any downstream model for recognition - almost like translating unfamiliar sparse keypoint representation (JKP) into the visual domain.
Some claims go beyond the evidence. "Matches the human baseline" holds only on NTU-CI, where the generator is trained on 14,063 in-distribution pairs and SFT reaches the same 80.4 with InternVL3-8B; there is no human baseline on NTU-CA or UTD. On UTD, absolute accuracy stays low (20.9–32.4),
About the human baseline, we agree with you. We will modify the claim and state that human evaluation is conducted only on cross-instance split. This is due to the costs involved on Amazon Mechanical Turk. We will clarify this in the final version of the paper.
several emergent abilities are shown only qualitatively.
We would like to clarify that our paper already provides three quantitative evaluations of generalization beyond JohanssonBench: (i) novel stimuli like silhouettes, outlines, and other combinations with keypoints; (Sec. 5.1); (ii) privacy-preserving event-camera inputs (Sec. 5.2); and (iii) generating co-speech gestures (Sec. 5.3). These results quantitatively demonstrate that RBI generalizes beyond the keypoint stimuli used during training. To further strengthen our evaluation, we also quantify shape perception on SpookyBench (previously qualitative in Fig. 19), where RBI correctly recovers shapes from noise in 18/28 cases (64.3%) verified by manual inspection.
No inference cost is reported either, although every test video requires a 19B video-generation pass. What is the inference time and GPU cost per video for RBI compared with SFT?
We compare the inference latency (seconds/video) on a single H200 GPU. While RBI incurs higher computational cost (~7.3 s; much of it goes into generating a 5–10 s video), it achieves substantially stronger generalization to unseen actions and datasets. Importantly, this additional computation also enables transfer to novel motion-based tasks, including event-camera perception and psychophysical stimuli, without task-specific training. We will include a discussion on this cost-accuracy-versatality trade-off in the final version of the paper.
| Model | Latency (s/video) ↓ | Cross-action (%) ↑ | Cross-dataset (%) ↑ | Skeleton AR | Event-camera | Psychophysics |
|---|---|---|---|---|---|---|
| InternVL3-8B | 0.45 | 10.8 | 2.6 | ✔ | ✘ | ✘ |
| + SFT | 0.45 | 44.4 | 17.0 | ✔ | ✘ | ✘ |
| + RBI (Ours) | 7.26 | 87.2 | 21.8 | ✔ | ✔ | ✔ |
Which NTU protocol defines the train/test split? Do NTU-CA test videos share actors and scenes with the generator's training data?
We use the original NTU RGB+D-60 dataset's cross-subject protocol (Shahroudy et al., 2016 [1]): 20 subjects for training and 20 for testing. NTU-CA test videos come from the 20 held-out subjects, so no actors appears in the generator's training data. Since the cross-subject protocol does not split across scenes, the same scene background may appear in train-test. But this is not a problem for RBI since it can freely generate novel scene backgrounds much beyond the train set backgrounds (see Fig. 3).
[1] NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysis. Amir Shahroudy, Jun Liu, Tian-Tsong Ng, Gang Wang. CVPR 2016.
Thanks for the author's rebuttal. After reading the results, I raise significant concerns about the availability of this system. First, achieving these modalities are expensive and time-consuming, especially the event-camera data. Second, the latency is very high, which is unacceptable to me.
Thank you for the response. We are not sure what "availability of this system" means.
- While it is true that the latency of our model is higher than that of a MLLM alone, the MLLM achieves 0% over many of the examples, so the lower latency is of little worth: e.g. an MLLM achieves 0% on the shape-subset of SpookyBench while our model achieves 64.3% with no training on the dataset.
- Generalization to event-camera data is just one application of RBI which we discover without any training - this data is not necessary for RBI.
Q1. Skeleton-based baselines
Thank you for the encouraging feedback. We are grateful that you acknowledge that the benchmark is well-designed and that RBI generalizes well. We address your remaining concerns below one by one.
Skeleton-based baselines
We acknowledge prior work on skeleton-based action recognition. We also would like to highlight a key difference with our setting. Usually, skeleton-based action recognition methods also implicitly use limb information (e.g., as edges connecting the joints) treating it as a graph (ST-GCN [4], ST-TR [5], SkateFormer [6]). This implies that even from a single-frame, one can recognize the pose. In contrast, our task, inspired by Johansson’s experiments, deliberately requires recognizing actions only from moving joints while using no limb information. Thus, in our case, it is ambiguous to predict action from a single frame.
Nevertheless, following your suggestion, we conducted experiments to compare specialized skeleton-based action recognition methods across all three splits. The key takeaway is that while specialized models perform well on simpler seen actions, RBI generalizes substantially better on unseen actions and dataset. Details are provided below.
Experimental setting. For cross-instance split (seen actions), we trained ST-GCN and PoseC3D. For cross-action and cross-dataset splits, we trained CLIP-like dual encoder methods for zero-shot skeleton action recognition. Since we need to test generalization to new actions, we implement two prior methods: SA-DVAE [1] and TDSM [2] developed for zero-shot skeleton-based action recognition (ZSAR). Both the methods align pre-trained skeleton features with a pre-trained text encoder (e.g., CLIP or Sentence-BERT) which enables zero-shot recognition. As the pre-trained skeleton encoder, we pick PoseC3D [3] (we only use its “Joints” variant which does not use limb information for a fair comparison) and use the CLIP’s pre-trained text encoder. We follow SA-DVAE and TDSM architectures and fine-tuning recipe as faithfully as possible and train on exactly the same splits as we did with RBI. For cross-dataset evaluation, we trained them on NTU-14K train set and for cross-action evaluation, we trained them on NTU-7K. Results are shown in the Table below.
Results. On the simpler cross-instance split (seen actions), specialized skeleton models do better than RBI. But note that these are specialized models that can only do one task whereas RBI is a general model and it can do much more than simply predict actions from keypoints (see Table 3, Table 4 and Fig 5). Moreover, on cross-action and cross-dataset splits, RBI clearly and substantially outperforms skeleton-based methods.
Finally, we would like to recall that our goal was not to solve skeleton action recognition. Instead we showed how we can adapt general foundation models on our psychology-inspired task while also demonstrating fascinating emergent capabilities. In contrast, skeleton-based action recognition methods are specialized models designed for a single task.
[1] SA-DVAE: Improving Zero-Shot Skeleton-Based Action Recognition by Disentangled Variational Autoencoders. Sheng-Wei Li, Zi-Xiang Wei, Wei-Jie Chen, Yi-Hsin Yu, Chih-Yuan Yang, Jane Yung-jen Hsu. ECCV 2024.
[2] Bridging the Skeleton-Text Modality Gap: Diffusion-Powered Modality Alignment for Zero-shot Skeleton-based Action Recognition. Jeonghyeok Do, Munchurl Kim. ICCV 2025.
[3] Revisiting Skeleton-based Action Recognition. Haodong Duan, Yue Zhao, Kai Chen, Dahua Lin, Bo Dai. CVPR 2022.
[4] Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition. Sijie Yan, Yuanjun Xiong, Dahua Lin. AAAI 2018.
[5] Skeleton-based Action Recognition via Spatial and Temporal Transformer Networks. Chiara Plizzari, Marco Cannici, Matteo Matteucci. CVIU 2020.
[6] SkateFormer: Skeletal-Temporal Transformer for Human Action Recognition. Jeonghyeok Do, Munchurl Kim. ECCV 2024.