acceptodds
Withdrawn from ICLR 2027

SIGMA: Skeleton Injection with Generative Motion Augmentation for Zero-Shot Skeleton-Based Action Recognition

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Zero-shot skeleton-based action recognition asks a model to classify actions it has never seen, from a textual description alone. The difficulty is the distance between the two modalities: a sentence and a sequence of joint coordinates share no common ground, and the sentence is the only evidence available for the categories to be predicted. Text-to-motion generation changes that. From the same description, these models now synthesize plausible movement, a second source of evidence, and project it into the modality the recognizer consumes. The raw generated skeletons cannot serve directly as supervision: the joint layout differs, body proportions and timing drift, two-person interactions cannot be produced at all, and quality varies sharply from one sample to the next. SIGMA turns that output into training material, retargeting and rescaling it onto the recorded format, composing two-body interactions from the action verb, and obtaining a well-structured, balanced, and diverse training subset. To narrow the gap between text and skeleton, we also propose refining the text by generating geometry-aware and more discriminative sentences to prevent confusion between similar action categories. To combine generated and real skeletons, we have designed a novel training objective that treats both types of samples as two populations rather than one, as a global shift remains. At inference time, we compensate for this shift in the generated latents, as the relative order between unseen categories is preserved. On NTU RGB+D 60 and NTU RGB+D 120, SIGMA sets a new state-of-the-art in zero-shot recognition on three of four splits, by up to 22 points, and reports the best generalized harmonic mean on the other two. Generated skeletons alone account for up to points of harmonic mean over an identical system using only textual descriptions.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
zero-shot action recognition, skeleton-based action recognition, text-to-motion generation, synthetic data augmentation, cross-modal alignment, human motion synthesis

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_BbjvOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes a new method for zero-shot skeleton action recognition. Specifically, it uses a pre-trained text-to-motion (T2M) model to generate diverse skeleton sequences for scalable training. Then, it employs LLMs to generate three types of textual descriptions for cross-modal alignment. Experimental results show that the proposed method achieves competitive performance.

Critical strengths
  1. The idea of synthesizing skeleton sequences rather than action features, as commonly adopted in prior works, is appreciated.
  2. The design of composing two-body interactions is interesting.
Critical weaknesses
  1. The presentation quality of the paper is poor, and the writing is extremely obscure and difficult to follow. There are numerous overly complex and lengthy sentences, and the paragraphs are not well organized, especially in the Introduction section, making it difficult for readers to grasp the motivation and overall storyline. Moreover, many relevant citations are missing in the Related Work section, and several technical details are not clearly explained.
  2. The experimental setting is unfair compared to prior works, as an additional pre-trained text-to-motion model is incorporated, which has been trained on large-scale action datasets (potentially including unseen actions). Thus, prior knowledge of unseen actions may have been leaked during training, which contradicts the requirements of the standard zero-shot learning setting. Additionally, scaling up the training samples for cross-modal alignment while comparing against prior methods trained with limited samples raises further concerns about fairness.
  3. The proposed method lacks novelty and meaningful technical contributions. LLM-based semantic description expansion has been widely explored in prior works, and the cross-modal alignment module is directly adopted from FS-VAE without modification. Thus, the proposed method appears to be a combination of existing techniques, and I cannot identify any new insights or substantial technical contributions throughout the paper.
  4. The experiments are not sufficiently comprehensive or thorough. Tables 1 and 2 lack comparisons with recent SOTA methods published in 2026. Meanwhile, only the NTU series datasets are used for evaluation, without considering the widely adopted PKU-MMD dataset. Additionally, the ablation studies do not evaluate the effectiveness of the generated skeleton sequences. Furthermore, the paper lacks qualitative analyses of the proposed method and provides limited new insights, as most of the findings have already been discussed in prior works.
Questions for authors
  1. What is the role of the set-level assignment method during inference? It appears to follow a transductive setting rather than the inductive setting typically adopted in standard zero-shot learning.
  2. In the generalized zero-shot setting, how does the proposed method prevent unseen actions from being incorrectly classified as seen actions? The motivation and technical justification behind Eq. (6) remain unclear, particularly regarding how it addresses this issue.
Reviewer_6Qa2OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

SIGMA uses generated skeleton sequences to learn about unseen action classes. It combines these sequences with detailed text descriptions and a modified alignment loss. The paper tests the method on NTU RGB+D 60 and 120 under ZSL and GZSL, and also reports a ZSL version that groups test samples before assigning labels.

Critical strengths

The use of text-to-motion generation for zero-shot skeleton-based action recognition is novel.

Critical weaknesses
  1. Training setup and practical value SIGMA uses unseen class labels to generate skeletons for alignment training (Sections 3.1 and 3.3). This use of class descriptions is allowed in generative ZSL, but I am not convinced that knowing the target classes in advance fits the claimed real-world uses. More broadly, I question why this task is needed on old laboratory datasets with simple action labels. The paper does not explain what real-world problem this protocol solves. Further gains seem to offer little beyond a competition for higher benchmark scores. Figure 4 shows existing ZSL methods passing unseen actions to a “Seen Classifier.” This is misleading. Prior methods already use skeleton–text matching or classifiers trained on generated unseen-class features, as in SynSE. The figure should explain how SIGMA’s test-set clustering differs from these approaches.

  2. Motion generation and missing comparisons MotionGPT3 uses a 263-dimensional HumanML3D format, while NTU uses 25-joint coordinates. Section 3.1 outlines the conversion, but the exact joint mapping and how well it handles differences between datasets are unclear. Arranging two independently generated motions with direction and distance rules does not show that the two people interact realistically. No other motion generator is tested. The experiments do not compare with Neuron (CVPR 2025), SkeletonContext (CVPR 2026), or Flora (CVPR Findings 2026). Neuron and SkeletonContext are not explicitly cited, and Flora is only briefly discussed. These gaps make it hard to judge what SIGMA adds to recent work. I found the Related Work and Method sections hard to follow. Several abbreviations, symbols, and hyperparameters need clearer explanations, and Figures 1 and 2 are not very helpful. The “(?)” citations in Sections 2.1 and 2.2 also make me suspect that some references or claims may have been hallucinated. This makes the related-work discussion difficult to trust.

Questions for authors
  1. What real-world problem does this protocol solve? Why should the target classes be known before alignment training, and do the baselines make the same assumption?

  2. How exactly are HumanML3D motions converted to the NTU format? What evidence shows that the converted motions and two-person interactions are realistic?

  3. Does the method also work with another motion generator? Which methods and sources did the authors mean in Figure 4 and the “(?)” citations?

  4. Why are Neuron, SkeletonContext, and Flora missing from the experiments? How does SIGMA differ from them?

Reviewer_AknyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SIGMA generates skeleton examples for unseen action classes, retargets and curates them, and combines them with context-aware descriptions to train a skeleton–text alignment model. It introduces population-specific training adjustments and optional transductive cluster assignment. Evaluation covers four NTU-60/120 splits under inductive and transductive ZSL and inductive GZSL.

Critical strengths
  1. Generating, retargeting, and curating motion sequences offers a coherent way to provide skeleton-domain training evidence for unseen classes.
  2. Context-aware descriptions increase average ZSL accuracy from 65.9% to 74.1% over independently written descriptions under the controlled setting.
  3. The inductive variant achieves 67.3% ZSL accuracy on NTU-60 48/12 and the highest reported GZSL harmonic mean on two splits.
Critical weaknesses
  1. Section 3.2 does not explain how unseen-class confusable sets are constructed, leaving a key input to the strongest semantic component insufficiently specified.
  2. Although Table 1 explicitly marks transductive results, it lacks controls applying the same clustering and assignment procedure to baseline representations.
  3. Key reproduction details, including exact model configurations, training settings, and the latent–text dimensional mapping in Eq. (4), are unspecified.
  4. The claimed 7.4-point GZSL harmonic-mean gain from skeleton injection is not substantiated by a matched comparison with an otherwise identical model without injection.
Questions for authors
  1. How are unseen-class confusable sets obtained without labeled unseen examples, and can you provide the prompts and resulting class lists?
  2. How do baseline representations and SIGMA without injection perform under the same transductive clustering and assignment procedure?
  3. Can you specify the model configurations, training schedule, and dimensional mapping between skeleton latents and text features in Eq. (4)?
  4. Can you provide matched GZSL results with and without injection to substantiate the claimed 7.4-point harmonic-mean improvement?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.