SIGMA: Skeleton Injection with Generative Motion Augmentation for Zero-Shot Skeleton-Based Action Recognition

Zero-shot skeleton-based action recognition asks a model to classify actions it has never seen, from a textual description alone. The difficulty is the distance between the two modalities: a sentence and a sequence of joint coordinates share no common ground, and the sentence is the only evidence available for the categories to be predicted. Text-to-motion generation changes that. From the same description, these models now synthesize plausible movement, a second source of evidence, and project it into the modality the recognizer consumes. The raw generated skeletons cannot serve directly as supervision: the joint layout differs, body proportions and timing drift, two-person interactions cannot be produced at all, and quality varies sharply from one sample to the next. SIGMA turns that output into training material, retargeting and rescaling it onto the recorded format, composing two-body interactions from the action verb, and obtaining a well-structured, balanced, and diverse training subset. To narrow the gap between text and skeleton, we also propose refining the text by generating geometry-aware and more discriminative sentences to prevent confusion between similar action categories. To combine generated and real skeletons, we have designed a novel training objective that treats both types of samples as two populations rather than one, as a global shift remains. At inference time, we compensate for this shift in the generated latents, as the relative order between unseen categories is preserved. On NTU RGB+D 60 and NTU RGB+D 120, SIGMA sets a new state-of-the-art in zero-shot recognition on three of four splits, by up to 22 points, and reports the best generalized harmonic mean on the other two. Generated skeletons alone account for up to $7.4$ points of harmonic mean over an identical system using only textual descriptions.