GestureLens: Generalized Learning Framework for IMU-based Human Gesture Recognition

IMU-based gesture interfaces are being increasingly adopted as efficient, accessible, and intuitive alternatives to traditional input methods, such as touchscreens and voice. However, current gesture recognition algorithms are tailored to work for specific devices (e.g., smartwatches vs. earbuds) or user populations (e.g., blind vs. sighted users), limiting their generalizability. In this paper, we design GestureLens, a generalized IMU-based gesture recognition framework that works across devices and populations with minimal training samples. To overcome the challenges and high cost of collecting large-scale labeled training data, GestureLens leverages readily available unlabeled human activity data. The GestureLens pipeline comprises two stages: (1) pre-training a motion representation model using abundant unlabeled human activity data, and (2) training a gesture classifier on top of the frozen representation using a small amount of labeled gesture data. For pre-training, we introduce a token-based strategy and embeddings that learn to identify and focus attention on the key motion signatures in the temporal data For classification, we design a text-guided classifier that can reliably differentiate between temporally or semantically similar gestures. We evaluate GestureLens across both hand gestures (captured through a smartwatch) and earbud gestures (captured through earbuds), using data collected from blind and sighted users. Across these diverse devices and user populations, GestureLens achieves an accuracy of 85%, across an average of 13 gesture classes using only 10% of labeled data for training. GestureLens significantly outperforms state-of-the-art self-supervised learning approaches and specialized gesture recognition models.