acceptodds
Under review as a conference paper at ICLR 2027

Recognition by Imagination: A Generative Approach to Form-from-Motion

Abstract

We study the problem of recognizing actions from a minimalist motion stimuli with no appearance cues. We frame this as an instantiation of Johansson’s famous experiments in 1973 on recognizing actions from a video showing a handful of moving keypoints against a black background. While this collection of keypoints seems to be spread arbitrarily in a single frame, their motion enables action perception. We establish that modern MLLMs are not able to represent such signals. The objective of this paper is then to enable video models to be sensitive to such subtle motion stimuli. To perceive this minimalist motion and recognize actions, we propose a novel two-step Recognition by Imagination (RBI) approach: (i) imagination: we fine-tune a video generation model to generate a realistic video solely from the keypoint stimuli, (ii) recognition: then, we employ a standard MLLM to classify the action. RBI significantly outperforms frontier MLLMs and shows superior generalization compared to supervised fine-tuning of open MLLMs. Interestingly, while RBI is fine-tuned on keypoint videos, it exhibits some surprising emergent capabilities. It works off-the-shelf with novel stimuli such as silhouettes, shape outlines, and even event-based camera inputs. Finally, we also show that the same model, fine-tuned only on keypoint videos, qualitatively generalizes to other psychophysical form-from-motion stimuli.

Primary area:
generative models
Keywords:
biological motion perception, multimodal LLMs, video generation models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_jmN5OpenReview ↗
Rating
2(−1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper builds JohanssonBench, which renders NTU-60 and UTD-MHAD videos as point-light keypoint (JKP) stimuli, and shows that frontier MLLMs reach roughly half of a human baseline on them. It proposes RBI: a LoRA-tuned LTX-2 (19B) generates an RGB video from the JKP input, and a frozen MLLM then classifies the action. RBI is compared with LoRA fine-tuning of MLLMs on cross-instance, cross-action and cross-dataset splits.

Critical strengths
  1. The benchmark is well designed: three splits with increasing distribution shift, an AMT human baseline (81%, Fleiss κ = 0.6), and a broad sweep of open and closed MLLMs. Showing that frontier models are near half of human level on point-light motion is a useful probe of how much MLLM action recognition relies on appearance.

  2. Analysis-by-synthesis generalizes better than fine-tuning. On unseen actions (NTU-CA), RBI averages 80.8 versus 47.6 for LoRA-SFT across four MLLMs, close to the RGB upper bound of 88.3; on UTD it reaches 24.8 versus 13.8. Off-the-shelf results on event-camera data and gestures support the idea beyond one setting.

Critical weaknesses
  1. The most natural baselines are missing. JKP is rendered from DWPose keypoints, so skeleton recognizers trained on the same 2D keypoints (e.g., ST-GCN or PoseC3D), and zero-shot skeleton methods for the cross-action split, are the obvious comparisons; only MLLM LoRA-SFT is used. No inference cost is reported either, although every test video requires a 19B video-generation pass.

  2. Some claims go beyond the evidence. "Matches the human baseline" holds only on NTU-CI, where the generator is trained on 14,063 in-distribution pairs and SFT reaches the same 80.4 with InternVL3-8B; there is no human baseline on NTU-CA or UTD. On UTD, absolute accuracy stays low (20.9–32.4), and several emergent abilities are shown only qualitatively.

I am not very clear why we need this representation for analysis. It's very rare and unconvincing for me.

Questions for authors
  1. How does a skeleton recognizer trained on the same DWPose 2D keypoints perform on all three splits, and how does a zero-shot skeleton method perform on NTU-CA?

  2. What is the inference time and GPU cost per video for RBI compared with SFT?

  3. Which NTU protocol defines the train/test split? Do NTU-CA test videos share actors and scenes with the generator's training data?

4 comments
AuthorsOpenReview ↗

Q1. Skeleton-based baselines

Thank you for the encouraging feedback. We are grateful that you acknowledge that the benchmark is well-designed and that RBI generalizes well. We address your remaining concerns below one by one.

Skeleton-based baselines

The most natural baselines are missing. JKP is rendered from DWPose keypoints, so skeleton recognizers trained on the same 2D keypoints (e.g., ST-GCN or PoseC3D), and zero-shot skeleton methods for the cross-action split, are the obvious comparisons; only MLLM LoRA-SFT is used. How does a skeleton recognizer trained on the same DWPose 2D keypoints perform on all three splits, and how does a zero-shot skeleton method perform on NTU-CA?

We acknowledge prior work on skeleton-based action recognition. We also would like to highlight a key difference with our setting. Usually, skeleton-based action recognition methods also implicitly use limb information (e.g., as edges connecting the joints) treating it as a graph (ST-GCN [4], ST-TR [5], SkateFormer [6]). This implies that even from a single-frame, one can recognize the pose. In contrast, our task, inspired by Johansson’s experiments, deliberately requires recognizing actions only from moving joints while using no limb information. Thus, in our case, it is ambiguous to predict action from a single frame.

Nevertheless, following your suggestion, we conducted experiments to compare specialized skeleton-based action recognition methods across all three splits. The key takeaway is that while specialized models perform well on simpler seen actions, RBI generalizes substantially better on unseen actions and dataset. Details are provided below.

Experimental setting. For cross-instance split (seen actions), we trained ST-GCN and PoseC3D. For cross-action and cross-dataset splits, we trained CLIP-like dual encoder methods for zero-shot skeleton action recognition. Since we need to test generalization to new actions, we implement two prior methods: SA-DVAE [1] and TDSM [2] developed for zero-shot skeleton-based action recognition (ZSAR). Both the methods align pre-trained skeleton features with a pre-trained text encoder (e.g., CLIP or Sentence-BERT) which enables zero-shot recognition. As the pre-trained skeleton encoder, we pick PoseC3D [3] (we only use its “Joints” variant which does not use limb information for a fair comparison) and use the CLIP’s pre-trained text encoder. We follow SA-DVAE and TDSM architectures and fine-tuning recipe as faithfully as possible and train on exactly the same splits as we did with RBI. For cross-dataset evaluation, we trained them on NTU-14K train set and for cross-action evaluation, we trained them on NTU-7K. Results are shown in the Table below.

Results. On the simpler cross-instance split (seen actions), specialized skeleton models do better than RBI. But note that these are specialized models that can only do one task whereas RBI is a general model and it can do much more than simply predict actions from keypoints (see Table 3, Table 4 and Fig 5). Moreover, on cross-action and cross-dataset splits, RBI clearly and substantially outperforms skeleton-based methods.

MethodBase modelModalitiesCross-instanceCross-actionCross-dataset
ST-GCNGCNsSkeleton89.1--
PoseC3D3D-CNNSkeleton92.9--
SA-DVAE (ECCV'24)3D-CNNSkeleton-text82.527.215.5
TDSM (ICCV'25)3D-CNNSkeleton-text64.250.411.4
MLLM (zero-shot)InternVL3-8BVideo-text7.410.82.6
+ SFTInternVL3-8BVideo-text80.444.417.0
+ RBI (Ours)InternVL3-8BVideo-text80.487.221.8

Finally, we would like to recall that our goal was not to solve skeleton action recognition. Instead we showed how we can adapt general foundation models on our psychology-inspired task while also demonstrating fascinating emergent capabilities. In contrast, skeleton-based action recognition methods are specialized models designed for a single task.


[1] SA-DVAE: Improving Zero-Shot Skeleton-Based Action Recognition by Disentangled Variational Autoencoders. Sheng-Wei Li, Zi-Xiang Wei, Wei-Jie Chen, Yi-Hsin Yu, Chih-Yuan Yang, Jane Yung-jen Hsu. ECCV 2024.

[2] Bridging the Skeleton-Text Modality Gap: Diffusion-Powered Modality Alignment for Zero-shot Skeleton-based Action Recognition. Jeonghyeok Do, Munchurl Kim. ICCV 2025.

[3] Revisiting Skeleton-based Action Recognition. Haodong Duan, Yue Zhao, Kai Chen, Dahua Lin, Bo Dai. CVPR 2022.

[4] Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition. Sijie Yan, Yuanjun Xiong, Dahua Lin. AAAI 2018.

[5] Skeleton-based Action Recognition via Spatial and Temporal Transformer Networks. Chiara Plizzari, Marco Cannici, Matteo Matteucci. CVIU 2020.

[6] SkateFormer: Skeletal-Temporal Transformer for Human Action Recognition. Jeonghyeok Do, Munchurl Kim. ECCV 2024.

AuthorsOpenReview ↗

Addressing the remaining questions

Here, we address the remaining questions.

I am not very clear why we need this representation for analysis. It's very rare and unconvincing for me.

We are happy to clarify the motivation and broader significance of our study on Johansson's keypoint representation. Our work is motivated by a fundamental question: do modern video models perceive biological motion from Johansson's sparse moving keypoints, as humans do? To measure this, we use Johansson's keypoints as testbed for probing pure temporal perception with no appearance information.

Broader significance. First, JohanssonBench tests for motion perception from sparse keypoints alone which can be indicate of real-world scenarios when appearance cues are unreliable, e.g., under low lighting, occlusion, or distant observations [1]. Second, beyond Johansson, RBI demonstrates the broader potential of generative reconstruction as an interpretable bridge between sparse motion and pretrained MLLMs. There are several benefits to the intermediate RGB generation:

  1. Generalization to novel tasks: Without any training, RBI is able to solve the psychophysical tests like the Hidden Bird Illusion (Fig. 5) or even work with novel stimuli like Dynamic-Vision Sensors (DVS).
  2. Interpretability: RGB output is human-interpretable rather than opaque latent features. As shown in Fig. 6 (newly added in the paper), we show that RBI, while generating RGB outputs, learns to map local keypoints accurately to human joints.
  3. Provides general-purpose visual interface to pretrained MLLMs: Generating RGB videos allows us to use any downstream model for recognition - almost like translating unfamiliar sparse keypoint representation (JKP) into the visual domain.

Some claims go beyond the evidence. "Matches the human baseline" holds only on NTU-CI, where the generator is trained on 14,063 in-distribution pairs and SFT reaches the same 80.4 with InternVL3-8B; there is no human baseline on NTU-CA or UTD. On UTD, absolute accuracy stays low (20.9–32.4),

About the human baseline, we agree with you. We will modify the claim and state that human evaluation is conducted only on cross-instance split. This is due to the costs involved on Amazon Mechanical Turk. We will clarify this in the final version of the paper.


several emergent abilities are shown only qualitatively.

We would like to clarify that our paper already provides three quantitative evaluations of generalization beyond JohanssonBench: (i) novel stimuli like silhouettes, outlines, and other combinations with keypoints; (Sec. 5.1); (ii) privacy-preserving event-camera inputs (Sec. 5.2); and (iii) generating co-speech gestures (Sec. 5.3). These results quantitatively demonstrate that RBI generalizes beyond the keypoint stimuli used during training. To further strengthen our evaluation, we also quantify shape perception on SpookyBench (previously qualitative in Fig. 19), where RBI correctly recovers shapes from noise in 18/28 cases (64.3%) verified by manual inspection.


No inference cost is reported either, although every test video requires a 19B video-generation pass. What is the inference time and GPU cost per video for RBI compared with SFT?

We compare the inference latency (seconds/video) on a single H200 GPU. While RBI incurs higher computational cost (~7.3 s; much of it goes into generating a 5–10 s video), it achieves substantially stronger generalization to unseen actions and datasets. Importantly, this additional computation also enables transfer to novel motion-based tasks, including event-camera perception and psychophysical stimuli, without task-specific training. We will include a discussion on this cost-accuracy-versatality trade-off in the final version of the paper.

ModelLatency (s/video) ↓Cross-action (%) ↑Cross-dataset (%) ↑Skeleton AREvent-cameraPsychophysics
InternVL3-8B0.4510.82.6✔✘✘
+ SFT0.4544.417.0✔✘✘
+ RBI (Ours)7.2687.221.8✔✔✔

Which NTU protocol defines the train/test split? Do NTU-CA test videos share actors and scenes with the generator's training data?

We use the original NTU RGB+D-60 dataset's cross-subject protocol (Shahroudy et al., 2016 [1]): 20 subjects for training and 20 for testing. NTU-CA test videos come from the 20 held-out subjects, so no actors appears in the generator's training data. Since the cross-subject protocol does not split across scenes, the same scene background may appear in train-test. But this is not a problem for RBI since it can freely generate novel scene backgrounds much beyond the train set backgrounds (see Fig. 3).

[1] NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysis. Amir Shahroudy, Jun Liu, Tian-Tsong Ng, Gang Wang. CVPR 2016.

Thanks for the author's rebuttal. After reading the results, I raise significant concerns about the availability of this system. First, achieving these modalities are expensive and time-consuming, especially the event-camera data. Second, the latency is very high, which is unacceptable to me.

Thank you for the response. We are not sure what "availability of this system" means.

  • While it is true that the latency of our model is higher than that of a MLLM alone, the MLLM achieves 0% over many of the examples, so the lower latency is of little worth: e.g. an MLLM achieves 0% on the shape-subset of SpookyBench while our model achieves 64.3% with no training on the dataset.
  • Generalization to event-camera data is just one application of RBI which we discover without any training - this data is not necessary for RBI.
Reviewer_FpHgOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces JohanssonBench to evaluate action recognition from sparse human keypoint motion, removing appearance cues from RGB videos. The authors show that current MLLMs struggle with these inputs compared to human annotators. To address this, they propose Recognition by Imagination (RBI), which fine-tunes a video generation model to reconstruct RGB videos from keypoint sequences, followed by a frozen MLLM for action classification.

Experiments show that RBI substantially improves recognition accuracy over direct supervised fine-tuning, particularly on unseen action classes. The authors also demonstrate generalization to other visual representations, including silhouettes, event-camera inputs, and gesture sequences.

Critical strengths
  1. Interesting problem formulation. Using Johansson's point-light stimuli provides a controlled setting to investigate whether MLLMs recognize actions from motion rather than relying on appearance cues. The benchmark also includes human evaluation, which helps establish that the stimuli contain recognizable action information.

  2. Strong cross-action generalization. RBI substantially outperforms supervised fine-tuning on unseen actions (80.8% vs. 47.6% on average). This is an interesting result suggesting that generative reconstruction can serve as an effective intermediate representation for downstream action recognition.

Critical weaknesses
  1. Unclear motivation and broader significance. While Johansson's keypoints are useful for probing biological motion perception, it is unclear why improving MLLM recognition on this particular representation is an important objective beyond the benchmark itself. The proposed method does not directly improve an MLLM's ability to understand motion, but instead converts sparse inputs into RGB videos that existing MLLMs can already recognize. Since the original RGB videos are available in standard action recognition settings, the practical benefit of this additional generation step is not sufficiently established.

  2. Lack of evaluation of motion fidelity. The main evaluation measures action classification accuracy, but does not directly assess whether generated videos faithfully preserve the input motion. For example, a model could generate a plausible hand-waving video while changing the acting hand, motion direction, or number of repetitions, yet still receive the correct action label. Therefore, high classification accuracy does not necessarily demonstrate accurate motion reconstruction or understanding. This distinction is particularly important given the paper's central claim about recovering motion information through imagination. While the appendix includes temporal sensitivity experiments, they do not directly establish motion fidelity.

Questions for authors
  1. Motion fidelity: Have the authors evaluated how faithfully the generated RGB videos preserve the original keypoint trajectories? For example, extracting keypoints from the generated videos and comparing them against the input trajectories could reveal errors in motion direction, repetition count, and temporal alignment. This would help clarify whether RBI actually preserves motion information or mainly generates plausible actions.

  2. Motivation and practical relevance: Beyond JohanssonBench, what is the main benefit of converting keypoint videos into RGB videos before recognition? Does the proposed approach provide advantages over simpler representations or direct recognition methods in practical settings?

  3. Human baseline: The reported 81% human accuracy is based on majority voting among three annotators per video. Could the authors also report individual annotator accuracy? This would help clarify how the human baseline should be interpreted when comparing against individual MLLMs.

3 comments
AuthorsOpenReview ↗

Q1. Motion fidelity

Thank you for the detailed review. We are glad that you find our problem interesting and RBI’s generalization strong. Here, we address your remaining concerns one by one.


Q1. Motion fidelity

Motion fidelity: Have the authors evaluated how faithfully the generated RGB videos preserve the original keypoint trajectories? For example, extracting keypoints from the generated videos and comparing them against the input trajectories could reveal errors in motion direction, repetition count, and temporal alignment. This would help clarify whether RBI actually preserves motion information or mainly generates plausible actions.

Thank you for the suggestions on evaluating motion fidelity. We measure motion fidelity and generation quality in two ways.

(1) Keypoint comparison. We conduct a quantitative comparison of keypoints on real vs. generated videos. Concretely, we obtain original keypoints using DWPose from a RGB video. Then, we use these keypoints as inputs to RBI to generate a RGB video. We then run DWPose on this generated video to obtain a new set of keypoints. We compare the two sets of keypoints using multiple metrics: (a) Normalized Mean Per Joint Position Error (N-MPJPE which ranges between [0, ∞) - lower is better); (b) Percentage of Correct Keypoints at a 0.2 threshold ([email protected] which ranges in [0, 1] - higher is better); (c) motion-direction cosine (compares direction of motion of keypoints; ranges in [-1, 1] - higher is better); (d) temporal alignment: mean absolute time shift that best aligns real and generated keypoint-velocity sequences (ranges in [0, 1000 ms]; lower is better). For this experiment, we select a subset of single-person action to avoid identity-discrepancy in two-person actions.

MetricRangeResult
N-MPJPE ↓[0, ∞)0.16
[email protected] ↑[0, 1]0.92
Motion direction (cosine similarity) ↑[-1, 1]0.69
Temporal alignment (ms) ↓[0, 1000]21

Low N-MPJPE (0.16) and high [email protected] (0.92) indicate strong preservation of joint positions, while the positive motion-direction similarity (0.69) suggests that RBI also substantially preserves the direction of movement indicating strong motion-fidelity. Finally, the temporal alignment is 21 ms (lower than sub-frame precision which is 33ms for 30 FPS) which confirms the strong alignment.

In Fig. 6 (newly added in the paper), we also qualitatively visualise the real and generated keypoint trajectories for a sample video.

(2) HumanScore. We also use the recently proposed HumanScore metric [7]. It is a systematic framework to evaluate the quality of human motions in AI-generated videos. Broadly, it evaluates kinematic plausibility, temporal stability, and biomechanical consistency. On original videos, we get an average HumanScore of 98.13% while that on RBI’s generated videos is 97.53%. This shows that generated videos reliably represent motion.


[7] HumanScore: Benchmarking Human Motions in Generated Videos. Tiange Xiang, Yusu Fang, Tian Tan, Narayan Schuetz, Scott Delp, Li Fei-Fei, Ehsan Adeli. NeurIPS 2026.

AuthorsOpenReview ↗

Q2. Motivation and practical relevance

Unclear motivation and broader significance. While Johansson's keypoints are useful for probing biological motion perception, it is unclear why improving MLLM recognition on this particular representation is an important objective beyond the benchmark itself. The proposed method does not directly improve an MLLM's ability to understand motion, but instead converts sparse inputs into RGB videos that existing MLLMs can already recognize. Since the original RGB videos are available in standard action recognition settings, the practical benefit of this additional generation step is not sufficiently established.

Beyond JohanssonBench, what is the main benefit of converting keypoint videos into RGB videos before recognition? Does the proposed approach provide advantages over simpler representations or direct recognition methods in practical settings?

Motivation. We clarify that our purpose is not to replace standard RGB-based action recognition settings. Our work is motivated by a fundamental question: do modern video models perceive biological motion from Johansson's sparse moving keypoints, as humans do? To measure this, we use Johansson's keypoints as testbed for probing pure temporal perception with no appearance information.

Broader significance. First, JohanssonBench tests for motion perception from sparse keypoints alone which can be indicate of real-world scenarios when appearance cues are unreliable, e.g., under low lighting, occlusion, or distant observations [1]. Second, beyond Johansson, RBI demonstrates the broader potential of generative reconstruction as an interpretable bridge between sparse motion and pretrained MLLMs. We list the benefits of intermediate RGB generation below.

Benefits of intermediate RGB generation. Our fine-tuning recipe forces the model to reconstruct appearance information solely from motion information. This yields the following benefits:

  1. Generalization to novel tasks: Without any training, RBI is able to solve the psychophysical tests like the Hidden Bird Illusion (Fig. 5) or even work with novel stimuli like Dynamic-Vision Sensors (DVS) (Fig. 5) or variants of human pose representation (Fig. 4).
  2. Interpretability: RGB output is human-interpretable rather than opaque latent features. As shown in Fig. 6 (newly added in the paper), we show that RBI, while generating RGB outputs, learns to map local keypoints accurately to human joints. We vary the LoRA guidance weight smoothly between γ∈[0.5,1.0]\gamma \in [0.5, 1.0] which lets us visualize influence of our RBI-finetuned recipe on the base model. As γ\gamma increases, we gradually see how the model converts keypoints into accurate human motion.
  3. Provides general-purpose visual interface to pretrained MLLMs: Generating RGB videos allows us to use any downstream model for recognition - almost like translating unfamiliar sparse keypoint representation (JKP) into the visual domain.

[1] Grossman, E.D. and Blake, R. Perception of coherent motion, biological motion and form-from-motion under dim-light conditions. Vision research, 1999.

AuthorsOpenReview ↗

Q3. Human baseline

Human baseline: The reported 81% human accuracy is based on majority voting among three annotators per video. Could the authors also report individual annotator accuracy? This would help clarify how the human baseline should be interpreted when comparing against individual MLLMs.

Thanks for the suggestion. Each example is annotated by k=3k=3 annotators. In total, our survey included K=39K=39 annotators with each labeling ∼4040 examples on average. As requested, we compute the mean per-annotator accuracy to be 70.7% (median 79.3%). We find large variance across annotatores due to some annotators labeling very few samples (11 of 39 label < 5 examples). Excluding these, we get mean per-annotator accuracy to be 77.0% (median 82.2%). The majority vote is robust to such outliers to a large extent. We will include these details in the Appendix.

We hope this addresses your concerns. We will be happy to engage in follow-up discussions. We thank you again for your time and effort.

Reviewer_7FbxOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates biological motion perception in MLLMs and introduces JohanssonBench to evaluate action recognition from sparse keypoint videos. The authors propose Recognition by Imagination (RBI), which generates RGB videos from keypoints using a fine-tuned video generation model and then performs recognition with a frozen MLLM. Experiments demonstrate improved cross-action and cross-dataset generalization over supervised fine-tuning, along with potential applications to other motion-based stimuli.

Critical strengths
  1. Interesting problem and benchmark. The paper introduces JohanssonBench to evaluate biological motion perception in MLLMs, with meaningful cross-instance, cross-action, and cross-dataset settings.

  2. Strong generalization results. RBI substantially outperforms supervised fine-tuning on unseen actions and datasets, demonstrating the potential of generative reconstruction for motion recognition.

  3. Broad experimental evaluation. The paper explores generalization to silhouettes, event-camera inputs, co-speech gestures, and other motion stimuli, demonstrating potential applications beyond the main benchmark.

Critical weaknesses
  1. Missing important baselines. The paper lacks comparisons with skeleton-based recognition methods, such as ST-GCN and skeleton transformers, which are natural alternatives for sparse keypoint inputs.

  2. Limited methodological novelty. RBI mainly combines existing video generation and recognition models. It remains unclear whether explicit RGB generation is necessary for the improvements.

  3. High computational cost. The approach relies on a 19B-parameter video generator followed by an MLLM, introducing considerable computational overhead without sufficient efficiency comparisons.

  4. Insufficient generalization analysis. Several generalization claims rely primarily on qualitative examples. Quantitative evaluation of motion fidelity and potential generative hallucinations is limited.

Questions for authors
  1. How does RBI compare with dedicated skeleton-based action recognition methods under cross-action and cross-dataset settings?

  2. Can the authors compare explicit RGB generation with direct keypoint representation learning or latent-space reconstruction to justify the generative design?

  3. How well do the generated videos preserve the original motion information, and how do generative hallucinations affect recognition accuracy?

  4. Can the authors provide end-to-end computational costs and compare efficiency against lightweight motion recognition methods?

4 comments
AuthorsOpenReview ↗

Q1. Skeleton-based baselines

Thank you for the detailed and constructive feedback. We are encouraged that you found the problem interesting and acknowledged RBI’s strong generalization. We address your remaining concerns step-by-step.


Q1. Skeleton-based baselines

Missing important baselines. The paper lacks comparisons with skeleton-based recognition methods, such as ST-GCN and skeleton transformers, which are natural alternatives for sparse keypoint inputs.

Q1: How does RBI compare with dedicated skeleton-based action recognition methods under cross-action and cross-dataset settings?

We acknowledge prior work on skeleton-based action recognition. We also would like to highlight a key difference with our setting. Usually, skeleton-based action recognition methods also implicitly use limb information (e.g., as edges connecting the joints) treating it as a graph (ST-GCN [4], ST-TR [5], SkateFormer [6]). This implies that even from a single-frame, one can recognize the pose. In contrast, our task, inspired by Johansson’s experiments, deliberately requires recognizing actions only from moving joints while using no limb information. Thus, in our case, it is ambiguous to predict action from a single frame.

Nevertheless, following your suggestion, we conducted experiments to compare specialized skeleton-based action recognition methods across all three splits. The key takeaway is that while specialized models perform well on simpler seen actions, RBI generalizes substantially better on unseen actions and dataset. Details are provided below.

Experimental setting. For cross-instance split (seen actions), we trained ST-GCN and PoseC3D. For cross-action and cross-dataset splits, we trained CLIP-like dual encoder methods for zero-shot skeleton action recognition. Since we need to test generalization to new actions, we implement two prior methods: SA-DVAE [1] and TDSM [2] developed for zero-shot skeleton-based action recognition (ZSAR). Both the methods align pre-trained skeleton features with a pre-trained text encoder (e.g., CLIP or Sentence-BERT) which enables zero-shot recognition. As the pre-trained skeleton encoder, we pick PoseC3D [3] (we only use its “Joints” variant which does not use limb information for a fair comparison) and use the CLIP’s pre-trained text encoder. We follow SA-DVAE and TDSM architectures and fine-tuning recipe as faithfully as possible and train on exactly the same splits as we did with RBI. For cross-dataset evaluation, we trained them on NTU-14K train set and for cross-action evaluation, we trained them on NTU-7K. Results are shown in the Table below.

Results. On the simpler cross-instance split (seen actions), specialized skeleton models do better than RBI. But note that these are specialized models that can only do one task whereas RBI is a general model and it can do much more than simply predict actions from keypoints (see Table 3, Table 4 and Fig 5). Moreover, on cross-action and cross-dataset splits, RBI clearly and substantially outperforms skeleton-based methods.

MethodBase modelModalitiesCross-instanceCross-actionCross-dataset
ST-GCNGCNsSkeleton89.1--
PoseC3D3D-CNNSkeleton92.9--
SA-DVAE (ECCV'24)3D-CNNSkeleton-text82.527.215.5
TDSM (ICCV'25)3D-CNNSkeleton-text64.250.411.4
MLLMInternVL3-8BVideo-text7.410.82.6
MLLM + SFTInternVL3-8BVideo-text80.444.417.0
RBI (Ours)InternVL3-8BVideo-text80.487.221.8

Finally, we would like to recall that our goal was not to solve skeleton action recognition. Instead we showed how we can adapt general foundation models on our psychology-inspired task while also demonstrating fascinating emergent capabilities. In contrast, skeleton-based action recognition methods are specialized models designed for a single task.


[1] SA-DVAE: Improving Zero-Shot Skeleton-Based Action Recognition by Disentangled Variational Autoencoders. Sheng-Wei Li, Zi-Xiang Wei, Wei-Jie Chen, Yi-Hsin Yu, Chih-Yuan Yang, Jane Yung-jen Hsu. ECCV 2024.

[2] Bridging the Skeleton-Text Modality Gap: Diffusion-Powered Modality Alignment for Zero-shot Skeleton-based Action Recognition. Jeonghyeok Do, Munchurl Kim. ICCV 2025.

[3] Revisiting Skeleton-based Action Recognition. Haodong Duan, Yue Zhao, Kai Chen, Dahua Lin, Bo Dai. CVPR 2022.

[4] Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition. Sijie Yan, Yuanjun Xiong, Dahua Lin. AAAI 2018.

[5] Skeleton-based Action Recognition via Spatial and Temporal Transformer Networks. Chiara Plizzari, Marco Cannici, Matteo Matteucci. CVIU 2020.

[6] SkateFormer: Skeletal-Temporal Transformer for Human Action Recognition. Jeonghyeok Do, Munchurl Kim. ECCV 2024.

AuthorsOpenReview ↗

Q2. Novelty and need for RGB generation

Limited methodological novelty. RBI mainly combines existing video generation and recognition models. It remains unclear whether explicit RGB generation is necessary for the improvements.

Can the authors compare explicit RGB generation with direct keypoint representation learning or latent-space reconstruction to justify the generative design?

We clarify that our goal was not to merely solve keypoint-based action recognition but to use that task to deeply investigate if modern videos models percieve Johansson's stimuli which serves as a proxy for pure temporal perception with no appearance. While we use existing models in RBI, the contribution lies in the formulation and demonstration of generation as an intermediate perceptual mechanism, rather than a novel backbone architecture. We list the benefits of intermediate RGB generation below.

Benefits of intermediate RGB generation. Our fine-tuning recipe forces the model to reconstruct appearance information solely from motion information. This yields the following benefits:

  1. Generalization to novel tasks: Without any training, RBI is able to solve the psychophysical tests like the Hidden Bird Illusion (Fig. 5) or even work with novel stimuli like Dynamic-Vision Sensors (DVS) (Fig. 5) or variants of human pose representation (Fig. 4).
  2. Interpretability: RGB output is human-interpretable rather than opaque latent features. As shown in Fig. 6 (newly added in the paper), we show that RBI, while generating RGB outputs, learns to map local keypoints accurately to human joints. We vary the LoRA guidance weight smoothly between γ∈[0.5,1.0]\gamma \in [0.5, 1.0] which lets us visualize influence of our RBI-finetuned recipe on the base model. As γ\gamma increases, we gradually see how the model converts keypoints into accurate human motion.
  3. Provides general-purpose visual interface to pretrained MLLMs: Generating RGB videos allows us to use any downstream model for recognition - almost like translating unfamiliar sparse keypoint representation (JKP) into the visual domain.
AuthorsOpenReview ↗

Q3. Computational cost

High computational cost. The approach relies on a 19B-parameter video generator followed by an MLLM, introducing considerable computational overhead without sufficient efficiency comparisons. Can the authors provide end-to-end computational costs and compare efficiency against lightweight motion recognition methods?

Thank you for the suggestion. We compare the inference latency (seconds/video) on a single H200 GPU. While RBI incurs higher computational cost (~7.3 s; much of it goes into generating a 5–10 s video), it achieves substantially stronger generalization to unseen actions and datasets. Importantly, this additional computation also enables transfer to novel motion-based tasks, including event-camera perception and psychophysical stimuli, without task-specific training. We will include a discussion on this cost-accuracy-versatality trade-off in the final version of the paper.

ModelLatency (s/video) ↓Cross-action (%) ↑Cross-dataset (%) ↑Skeleton AREvent-cameraPsychophysics
SA-DVAE0.0327.215.5✔✘✘
TDSM0.0450.411.4✔✘✘
InternVL3-8B0.4510.82.6✔✘✘
+ SFT0.4544.417.0✔✘✘
+ RBI (Ours)7.2687.221.8✔✔✔
AuthorsOpenReview ↗

Q4. Generalization and motion-fidelity

Insufficient generalization analysis. Several generalization claims rely primarily on qualitative examples. Quantitative evaluation of motion fidelity and potential generative hallucinations is limited.

We would like to clarify that our paper already provides three quantitative evaluations of generalization beyond JohanssonBench: (i) novel stimuli like silhouettes, outlines, and other combinations with keypoints; (Sec. 5.1); (ii) privacy-preserving event-camera inputs (Sec. 5.2); and (iii) generating co-speech gestures (Sec. 5.3). These results quantitatively demonstrate that RBI generalizes beyond the keypoint stimuli used during training. To further strengthen our evaluation, we also quantify shape perception on SpookyBench (previously qualitative in Fig. 19), where RBI correctly recovers shapes from noise in 18/28 cases (64.3%) verified by manual inspection.

Quantitative evaluation of motion fidelity and potential generative hallucinations is limited. How well do the generated videos preserve the original motion information, and how do generative hallucinations affect recognition accuracy?

We measure motion fidelity and generation quality in two ways.

(1) Keypoint comparison. We conduct a quantitative comparison of keypoints on real vs. generated videos. Concretely, we obtain original keypoints using DWPose from a RGB video. Then, we use these keypoints as inputs to RBI to generate a RGB video. We then run DWPose on this generated video to obtain a new set of keypoints. We compare the two sets of keypoints using multiple metrics: (a) Normalized Mean Per Joint Position Error (N-MPJPE which ranges between [0, ∞) - lower is better); (b) Percentage of Correct Keypoints at a 0.2 threshold ([email protected] which ranges in [0, 1] - higher is better); (c) motion-direction cosine (compares direction of motion of keypoints; ranges in [-1, 1] - higher is better). For this experiment, we select a subset of single-person action to avoid identity-discrepancy in two-person actions.

MetricRangeResult
N-MPJPE ↓[0, ∞)0.16
[email protected] ↑[0, 1]0.92
Motion direction (cosine similarity) ↑[-1, 1]0.69

Low N-MPJPE (0.16) and high [email protected] (0.92) indicate strong preservation of joint positions, while the positive motion-direction similarity (0.69) suggests that RBI also substantially preserves the direction of movement indicating strong motion-fidelity.

(2) HumanScore. We also use the recently proposed HumanScore metric [7]. It is a systematic framework to evaluate the quality of human motions in AI-generated videos. Broadly, it evaluates kinematic plausibility, temporal stability, and biomechanical consistency. On original videos, we get an average HumanScore of 98.13% while that on RBI’s generated videos is 97.53%. This shows that generated videos reliably represent motion.

We hope this addresses your concerns. We will be happy to engage in follow-up discussions. We thank you again for your time and effort.


[7] HumanScore: Benchmarking Human Motions in Generated Videos. Tiange Xiang, Yusu Fang, Tian Tan, Narayan Schuetz, Scott Delp, Li Fei-Fei, Ehsan Adeli. NeurIPS 2026.

Comments on the paper

1 comment
Area_Chair_nHvvOpenReview ↗

Discussion between Authors and Reviewers

Dear Reviewers,

The authors have submitted their rebuttal. Please read it and discuss with the authors via posted messages if necessary.

Note that the author-reviewer discussion phase ends on November 18 at 11:59 pm AoE. It is appreciated if you could share your thoughts as soon as possible, so the authors have enough time to clarify any remaining concerns.

Thank you!

Best, Your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.